Releases | Open Source Science

What's Changed

all: bump golang.org/x deps by @sbinet in https://github.com/go-hep/hep/pull/1026
all: remove gioui dependency by @sbinet in https://github.com/go-hep/hep/pull/1029
groot: bump to ROOT-6.34.04 by @sbinet in https://github.com/go-hep/hep/pull/1028
groot/rtree: testing ROOT/C++ by @sbinet in https://github.com/go-hep/hep/pull/1031
ci: improve run-tests command line interface by @sbinet in https://github.com/go-hep/hep/pull/1033
all: bump deps by @sbinet in https://github.com/go-hep/hep/pull/1035
xrootd/xrdio: ensure File implements io/fs.File by @sbinet in https://github.com/go-hep/hep/pull/1036
sliceop/f64s: remove by @sbinet in https://github.com/go-hep/hep/pull/1037
all: modernize Go usage by @sbinet in https://github.com/go-hep/hep/pull/1038
Mig codeberg deps by @sbinet in https://github.com/go-hep/hep/pull/1040
all: migrate to math/rand/v2 by @sbinet in https://github.com/go-hep/hep/pull/1027
fit: export Func1D.Hessian by @sbinet in https://github.com/go-hep/hep/pull/1041
fit: add Poly to fit polynomials by @sbinet in https://github.com/go-hep/hep/pull/1042
hplot: add pre/mid/post-steps kinds to S2D by @sbinet in https://github.com/go-hep/hep/pull/1043
all: use codeberg.org/raw instead of github.com/raw by @sbinet in https://github.com/go-hep/hep/pull/1044
ci: migrate to woodpecker by @sbinet in https://github.com/go-hep/hep/pull/1046
ci: add gitlab-ci support by @sbinet in https://github.com/go-hep/hep/pull/1045
all: bump x/deps by @sbinet in https://github.com/go-hep/hep/pull/1047

Full Changelog: https://github.com/go-hep/hep/compare/v0.36.0...v0.37.0

Scientific Software - Peer-reviewed - Go
Published by sbinet about 1 year ago

What's Changed

all: bump Go-1.23, drop Go-1.21 by @sbinet in https://github.com/go-hep/hep/pull/1016
groot/{riofs,rtree}: expose API to customize key-compression by @sbinet in https://github.com/go-hep/hep/pull/1017
groot/riofs: align default compression settings w/ ROOT by @sbinet in https://github.com/go-hep/hep/pull/1021

Full Changelog: https://github.com/go-hep/hep/compare/v0.35.0...v0.36.0

Scientific Software - Peer-reviewed - Go
Published by sbinet over 1 year ago

What's Changed

all: bump x/net@v0.17.0 and klauspost/compress@v1.17.1 by @sbinet in https://github.com/go-hep/hep/pull/1000
cmd/root2csv: add support for converting Graph{,Errors} to CSV by @sbinet in https://github.com/go-hep/hep/pull/1001
groot: add read support for TCanvas by @sbinet in https://github.com/go-hep/hep/pull/1002
groot: streamline serialization header extraction by @sbinet in https://github.com/go-hep/hep/pull/1003
all: bump x/crypto@v0.17.0, sbinet/npyio@v0.8.0 and klauspost/compress@v1.17.4 by @sbinet in https://github.com/go-hep/hep/pull/1005
all: bump to ROOT-6.30/02 by @sbinet in https://github.com/go-hep/hep/pull/1006
groot: add support for TScatter by @sbinet in https://github.com/go-hep/hep/pull/1007
Go1.22 by @sbinet in https://github.com/go-hep/hep/pull/1008
groot: add handling of top-level key with string payload by @sbinet in https://github.com/go-hep/hep/pull/1009
all: bump deps epok@v0.5.0, npyio@v0.9.0 and x/{crypto,sync,tools,net,sys} by @sbinet in https://github.com/go-hep/hep/pull/1012
all: adapt tests and floating-point precision for Mac-Silicon by @sbinet in https://github.com/go-hep/hep/pull/1013
all: bump to ROOT-6.32/00 by @sbinet

Full Changelog: https://github.com/go-hep/hep/compare/v0.34.1...v0.35.0

Scientific Software - Peer-reviewed - Go
Published by sbinet almost 2 years ago

What's Changed

groot/rhist: implement RSlicer for TGraph{,Errors,AsymErrors,MultiErrors} by @sbinet in https://github.com/go-hep/hep/pull/996
all: apply loopvar fixups by @sbinet in https://github.com/go-hep/hep/pull/997
all: bump epok@v0.4.0, x/{image,net,sync,tools} by @sbinet in https://github.com/go-hep/hep/pull/999

Full Changelog: https://github.com/go-hep/hep/compare/v0.34.0...v0.34.1

Scientific Software - Peer-reviewed - Go
Published by sbinet over 2 years ago

Go-HEP - Release v0.34.0

Release v0.34.0 is out of the oven.

This release contains no major API breakage, but all around improvements and bug fixes.

hplot

hplot/vgop: a new package to implement JSON serialization of gonum/plot canvases

go p := hplot.New() // ... err := hplot.Save(p, 10*vg.Centimeter, 20*vg.Centimeter, "plot.json")

hplot is now using gonum/plot@v0.14.0 which has the necessary infrastructure to get nicer timeseries axes. See the provided examples (here) for the hplot.Ticks type.
hplot has gained some basic capability to display the legend (a color palette with numbers) of a 2D plot. See the provided example.
added a convenience forward function for gonum/plot/plotter.NewLinePoints (as hplot.NewLinePoints)

groot

added support for ROOT 6.28/04
groot/rjson: a new package to implement JSROOT-compatible JSON serialization of groot's histogram types (rhist.H{1,2}x)

go h, err := f.Get("h1d") raw, err := rjson.Marshal(h)

See the complete example here.

groot/rtree is now able to read trees created by Geant4 in multithreaded mode (see here for more details).
added support for the "new" TLeafG branch.

Changelog

6e9be99e all: bump gioui@v0.30, x/{crypto,exp,image,text,tools}@latest
970796df hplot{,vgop}: add support for Save("foo.json")
e1cbaa61 hplot/vgop: first import
301b96c6 groot/rjson: first import
d280b3c9 groot: implement RSlicer for TH{1,2}x
8795baec groot/rbytes: introduce RSlicer interface
8631271d cmd/root2arrow: use latest arrow-cat for tests
1d490da0 cmd/root2npy: use internal/diff
0e69116b groot: add tests for TLeafG
85fd5fd9 groot: add support for TLeafG
7cb2fac8 groot: bump to ROOT 6.28/04
548d277c groot/rtree: handle parallel-merged TBaskets
404ed51c ci: bump staticcheck@2023.1.5
d41b03d0 hplot: add NewLinePoints forward
b6113fad hplot: add timeseries examples with epok
948b0093 hplot: add automatic legend creation to H2D
d26c986b hplot: add support for legend to Figure
02946662 hplot: forward plot.Legend
d0eea6ab fwk: bump to gonuts/commander@v0.4.1
fa91d5d0 all: bump Gonum@v0.14.0, Gioui@v0.2.0
233c6a47 all: bump Go-1.21, drop Go-1.19
d85e15f7 groot/cmd/root-gen-streamer: use an in-memory buffer for generation
c00ff1a0 hplot: update examples
8c9cc408 hplot: update embedmd invocation
673ab4c5 ci: improve mk-release to setup symlinks

Scientific Software - Peer-reviewed - Go
Published by sbinet over 2 years ago

Go-HEP - Release v0.33.0

Release v0.33.0 is out of the oven.

This release contains no major API breakage, but all around improvements and bug fixes.

cmd

cmd/hepmc2root: is a new command to automatically convert HepMC2 ASCII files to flat n-tuples ROOT files.

``` $> hepmc2root -h hepmc2root converts a HepMC2 ASCII file into a ROOT file and (flat) tree.

Usage: hepmc2root [OPTIONS] hepmc.ascii

Example:

$> hepmc2root ./hepmc.ascii $> hepmc2root -o out.root -t mytree ./hepmc.ascii

Options: -o string path to output ROOT file name (default "out.root") -t string name of the output tree (default "tree") ```

hplot

hplot has been migrated to the new Gio API, and needs Gonum/plot@v0.13.0.

Slight differences are to be expected in the rendering of plots. With the new Gio version we are using, you may need development headers for Vulkan on Linux. See the installation instructions of Gio for more details: Gio Installation

fix a bug in hplot.TiledPlot.Plot(i, j int) where the *hplot.Plot returned by Plot(i,j) was actually the conjugate of what the documentation specified. Now, Plot(i,j) returns the plot as documented.

sliceop

introduce a generic sliceop.Resize that can reuse []T capacity
introduce generic versions of sliceop.Take and sliceop.Find

Changelog

Hepmc2root by @sbinet in https://github.com/go-hep/hep/pull/962
groot/{riofs,rtree}: de-duplicate automatically generated This-streamers by @sbinet in https://github.com/go-hep/hep/pull/965
cmd/hepmc2root: improve documentation by @sbinet in https://github.com/go-hep/hep/pull/967
cmd/hepmc2root,hepmc{,/rootcnv}: implement HepMC-ROOT roundtrip by @sbinet in https://github.com/go-hep/hep/pull/968
internal/diff: first import by @sbinet in https://github.com/go-hep/hep/pull/969
all: 2023 is the year of the Gopher by @sbinet in https://github.com/go-hep/hep/pull/970
cmd/lhef2hepmc: add -keep-all flag to keep intermediaries by @sbinet in https://github.com/go-hep/hep/pull/971
Hepmc3 by @sbinet in https://github.com/go-hep/hep/pull/972
Go 1.20 by @sbinet in https://github.com/go-hep/hep/pull/973
ci: update gha-actions by @sbinet in https://github.com/go-hep/hep/pull/976
ci: use Go cache from setup-go by @sbinet in https://github.com/go-hep/hep/pull/977
hplot: fix TiledPlot.Plot to behave as documented by @sbinet in https://github.com/go-hep/hep/pull/978
all: update to latest Gio by @sbinet in https://github.com/go-hep/hep/pull/979

Full Changelog: https://github.com/go-hep/hep/compare/v0.32.1...v0.33.0

Scientific Software - Peer-reviewed - Go
Published by sbinet about 3 years ago

Go-HEP - Release v0.32.1

What's Changed

groot/riofs: provide a stable top-dir to Walk by @sbinet in https://github.com/go-hep/hep/pull/960

Full Changelog: https://github.com/go-hep/hep/compare/v0.32.0...v0.32.1

Scientific Software - Peer-reviewed - Go
Published by sbinet over 3 years ago

Go-HEP - Release v0.32.0

Release v0.32.0 is out of the oven.

This release contains no major API breakage, but all around improvements and bug fixes.

This release drops support for Go-1.17 (and earlier) and is the first release with the introduction of a few generics-based APIs.

Thanks to Olivier Mengué (@dolmen) there were a couple of house cleaning changes that have been applied (as part of the CERN+Google Hackathon). Thanks again Olivier.

There has been some work on RNTuple front, but nothing that can be released yet.

brio

General clean-ups by Olivier M.

fwk

General clean-ups by Olivier M.

groot

riofs: add generic Get[T] to retrieve ROOT objects
riofs: export in-memory read-only RMemFile ROOT file

hbook

General clean-ups by Olivier M.

lhef

General clean-ups by Oliver M. Olivier also reduced the amount of memory allocations in lhef.NewDecoder.

sliceop

introduce sliceop and generics-based functions that mimick what sliceop/f64s did
implement sliceop/f64s in terms of sliceop

Changelog

c8c6fbf9 fmom: update for new gonum.org/v1/gonum@v0.12.0/r3 API
3e7183c9 all: update gonum@v0.12, gokrb5@v8.4.3, compress@v1.15.9, lz4@v4.1.15, npyio@v0.7.0 and go/x
535619c2 ci: add Go-1.19
c6ebf251 all: use git.sr.ht/~sbinet/go-arrow@v0.2.0 fork
770e2dc1 all: bump x/tools@v0.1.11
ece275b1 groot: fix go:generate portability for root-gen-type
0c1ce510 brio-gen: cleanup files produced by test on test success
8c3e3fa2 ci: bump actions/{setup-go,cache,checkout}/v3
fafbeb67 all: bump yaml@v3.0.1
ff19fc7b groot: document interop model with ROOT/C++
0eeae227 groot/rmeta: regnerate stringer
f9b43650 groot/{cmd/root-gen-type,rdict}: add support for generating RVec types
dc2dcf63 groot/{rcmd,rdict}: add support for reading ROOT::VecOps::RVec
fd8c2dc4 cmd/lhef2hepmc: improve warning message and display event number
f979a1cb all: apply gofmt-1.19 formating rules
2d8ce49a fwk/internal/fwktest: apply staticcheck cosmetic change
f81a84e5 fwk: gofmt
f596c97c fwk: manual fix of the last reference of pkg testdata
d4ec96b7 fwk: fix references to pkg testdata
c04eef86 fwk: rename pkg ./testdata to ./internal/fwktest
07f22fc6 sliceop: use errors.New for constant errors
40700b82 hbook: regenerate files produced by brio-gen
25b579b8 hbook: fix go:generate to not require to install brio-gen
7fe81f63 brio-gen: fix generated code to follow the Go standard
3396f5f1 brio-gen: add .gitignore for temporary files produced by tests
19923699 lhef: reduce allocs on event decoding
9c5c5662 lhef: more Decoder tests
cbb49350 lhef: reduce allocs in NewDecoder
163d127b lhef: use errors.New instead of fmt.Errorf for constants
020835d7 cmd/root2yoda: removes unreacheable statement
cf9adbb1 ci: use codecov-action@v2
474d07ad groot/riofs: export in-memory read-only RMemFile ROOT file
bb8ad3d4 groot/{cmd/root-gen-type,riofs}: add testdata for base+derived classes
c52e43d3 groot/rdict: correctly handle generation of marshaling for base classes
7f90bc7e all: bump x/crypto, x/exp and x/sys
4f927c35 sliceop: improve performances of Take
c2ec83ed sliceop{,/f64s}: introduce generics sliceop
582dce6c groot/riofs: add generic Get[T] to retrieve ROOT objects
ccbaeb33 all: apply staticcheck fixes
e93f8a17 ci: use staticcheck instead of golangci-lint
7a3ff60c ci: reduce git-checkout depth
ddf2dc75 all: drop Go-1.17

What's Changed

all: drop Go-1.17 by @sbinet in https://github.com/go-hep/hep/pull/929
groot/riofs: add generic Get[T] to retrieve ROOT objects by @sbinet in https://github.com/go-hep/hep/pull/930
sliceop{,/f64s}: introduce generics sliceop by @sbinet in https://github.com/go-hep/hep/pull/931
all: bump x/crypto, x/exp and x/sys by @sbinet in https://github.com/go-hep/hep/pull/932
Groot gen type base by @sbinet in https://github.com/go-hep/hep/pull/933
groot/riofs: export in-memory read-only RMemFile ROOT file by @sbinet in https://github.com/go-hep/hep/pull/935
ci: use codecov-action@v2 by @sbinet in https://github.com/go-hep/hep/pull/936
removes unreacheable statement by @chavacava in https://github.com/go-hep/hep/pull/928
lhef: use errors.New instead of fmt.Errorf for constants by @dolmen in https://github.com/go-hep/hep/pull/937
lhef: reduce number of allocs in Decoder by @dolmen in https://github.com/go-hep/hep/pull/938
brio-gen: follow Go standard for generated files by @dolmen in https://github.com/go-hep/hep/pull/939
sliceop: use errors.New for constant errors by @dolmen in https://github.com/go-hep/hep/pull/940
fwk: rename package testdata to internal/fwktest by @dolmen in https://github.com/go-hep/hep/pull/942
Staticcheck fix by @sbinet in https://github.com/go-hep/hep/pull/943
cmd/lhef2hepmc: improve warning message and display event number by @sbinet in https://github.com/go-hep/hep/pull/944
Groot vecops by @sbinet in https://github.com/go-hep/hep/pull/945
groot/rmeta: regnerate stringer by @dolmen in https://github.com/go-hep/hep/pull/946
groot: document interop model with ROOT/C++ by @sbinet in https://github.com/go-hep/hep/pull/948
all: bump yaml@v3.0.1 by @sbinet in https://github.com/go-hep/hep/pull/949
ci: bump actions/{setup-go,cache,checkout}@v3 by @sbinet in https://github.com/go-hep/hep/pull/950
brio-gen: cleanup files produced by tests on test success by @dolmen in https://github.com/go-hep/hep/pull/951
groot: fix go:generate portability for root-gen-type by @dolmen in https://github.com/go-hep/hep/pull/952
Go x tools by @sbinet in https://github.com/go-hep/hep/pull/953
ci: add Go-1.19 by @sbinet in https://github.com/go-hep/hep/pull/954
all: update gokrb5@v8.4.3, compress@v1.15.9, lz4@v4.1.15, npyio@v0.7.… by @sbinet in https://github.com/go-hep/hep/pull/957

New Contributors

@chavacava made their first contribution in https://github.com/go-hep/hep/pull/928
@dolmen made their first contribution in https://github.com/go-hep/hep/pull/937

Full Changelog: https://github.com/go-hep/hep/compare/v0.31.0...v0.32.0

Scientific Software - Peer-reviewed - Go
Published by sbinet over 3 years ago

Go-HEP - Release v0.31.1

Full Changelog: https://github.com/go-hep/hep/compare/v0.31.0...v0.31.1

c31af820 groot/{cmd/root-gen-type,riofs}: add testdata for base+derived classes
8883629d groot/rdict: correctly handle generation of marshaling for base classes

Scientific Software - Peer-reviewed - Go
Published by sbinet about 4 years ago

Go-HEP - Release v0.31.0

Release v0.31.0 is out of the oven.

This release contains no major API breakage, but all around improvements and bug fixes.

This release drops support for Go-1.16 (and earlier).

groot

bump to ROOT-6.26/00
implement a riofs Reader over HTTP(s), instead of downloading the whole file locally
improve performances of rcmd.Dump(tree)
add support for TMultiGraph
add support for TDatime
add support for TGraphMultiErrors
add support for TProfile{,2D}
add support for TEfficiency
add support for TConfidenceLevel, TLimit{,DataSource}
add support for TF1 and TFormula
add support for reading std::vector<T, myalloc<T>>
introduced a new package groot/rnpy, to ease ROOT/NumPy conversions (like groot/rarrow does for Arrow)
improve memory usage of root2npy, by materializing into memory only one column at a time
streamline C++ templates parsing for STL containers (vector,deque,map,unordered_{,multi}{map,set}, pair, ...)

xrootd

improve the read performances 10-fold, using a finer-grained lock to protect session IDs

AOB

That's all for today. Last release we said there would probably be some work on the RNTuple front. We didn't lie, but this didn't happen in the form we thought it would. A surprise is in the works... stay tuned.

Next cycle will probably see some work on the RNTuple front, and that work appear in main.

Changelog

b02724a1 all: bump x/crypto
6c359638 all: drop old +build foo stanza, use //go:build
24817699 all: bump klauspost/compress@v1.15.1 and pierrec/lz4@v4.1.14
59c6c675 all: bump gonum/gonum@v0.11.0, gonum/plot@v0.11.0
c6f7c48c all: drop Go-1.16, add Go-1.18
7909cd7d groot/internal/httpio: add a pool of http.Request
0e0b9931 groot/riofs/plugin/http: first stab at a caching+concurrent http-reader
53a256f5 groot/internal/httpio: first import
e6d5f90c groot: bump to ROOT-6.26/00
6f88941b xrootd: improve read performances 10-fold
c38575f0 groot/rcmd: improve Dump tree performances
ca83c453 groot/{internal/rtests,riofs}: better ACliC handling
e453fa7c groot/{rcmd,rhist,rvers}: add support for TMultiGraph
17474123 groot/{rbase,riofs,rvers}: add support for TDatime
0034b01b groot/internal/rtests: make sure macros are run thru ACliC
0bd7c62e groot/{rcmd,riofs,rmeta}: add support for read std::vector>
551dabd7 groot/{rcmd,riofs}: add tests for more STL containers
f1aa9058 groot/rtree: add rvar dispatch for map[K]V
b4678544 groot/rdict: add r/w-streamer+type handling for TString
480971d2 groot/rdict: handle TString streaming in collections
22da5159 groot/rdict: use []T for disk storage of std::set
34d6347f groot/rdict: proper handling of 'This' streamers
0a85f502 groot: introduce rbytes.Header
e16891db groot/rcmd: add tests for POD data files
2a344611 groot/rdict: generate RVersioner if not implemented
d23d2f42 groot/{rdict,rmeta}: introduce CxxTemplate, streamline template parsing
be7ddfa6 groot/rnpy: new package to ease ROOT-Tree/NumPy conversion
37a9ac0e cmd/root2npy: improve memory usage
04a2d5fc groot: streamline r/w buffer ops
1b25234f groot/{rdict,rhist,rvers}: add r/w support for TGraphMultiErrors
73b95060 groot/{rhist,riofs,rvers}: add r/w support for TProfile{,2D}
449e81e7 groot/rdict: add initial support for enums in genGoType
e6f8d63d groot/{rcmd,rdict,rhist,rvers}: add r/w support for TEfficiency
929ec73d groot/rhist: add write support to TF1 and TFormula
129831f6 groot/{rcmd,rdict,rhist,riofs,rvers}: add r/w support for TConfidenceLevel and TLimit{,DataSource}
ffc817e8 root/rbytes: add WriteStdVectorF64
e488e92a root/rbytes: add WriteObject
fa6827ea groot/{rdict,rhist,rvers}: add initial read support for TF1 and TFormula
d8d3eb53 groot/rbytes: add RBuffer.ReadStdVectorI32
17373164 groot/rbytes: add RBuffer.ReadStdVectorF64
ad66388a cmd/root2{arrow,csv,npy}: use riofs.Dir to retrieve trees

Scientific Software - Peer-reviewed - Go
Published by sbinet about 4 years ago

Go-HEP - Release v0.30.1

Release v0.30.1 is out of the oven.

This release contains no major API breakage, but all around improvements and bug fixes.

fwk

updated fwk-app to use the latest gonuts/commander version that provides auto-completion of commands, flags and sub-commands.

groot

add support for TBranchObject and TLeafObject
add support for TLorentzVector (in root-dump and friends)
add support for TNtuple and TNtupleD (in root-dump and friends)

hplot

fixed a race in hplot.New where the global variable from gonum/plot was being modified without a critical section. This race appeared when one would create multiple hplot.Plot concurrently.

sliceop

improved performances of sliceop/f64s.Take. Do note that sliceop/f64s will probably be rewritten in terms of a generic implementation (and deprecated) once Go-1.18 is generally available.

AOB

That's all for today. Next cycle will probably see some work on the RNTuple front.

Changelog

76e7c865 all: bump npyio@v0.6.0
baa058b5 groot/{rdict,rtree,rvers}: bump to ROOT-6.24/06
bc000321 all: bump peterh/liner, x/crypto, x/image, x/tools and ql
b57330a1 groot/{rcmd,riofs,rtree,rvers}: add rtree.Reader support for TNtuple{,D}
24ff587b all: bump klauspost/compress@v1.14.2
d1e8a6fe all: 2022 is the year of the Gopher
505ab934 fwk/cmd/fwk-app: use gonuts/commander@v0.3.x
c5711342 hplot: fix race in hplot.New on gonum/plot.DefaultFont
eaeb641a groot/rphys: test RStreamer with Bind+Unmarshaler for TLorentzVector
679c8867 groot/rphys: make Vector{2,3} implement fmt.Stringer
7851644b groot/rphys: make LorentzVector implement fmt.Stringer
07e2951c groot/rdict: leverage rbytes.{Unm,M}arshaler implementation in Bind
e370fced groot/riofs: add generation of TLorentzVector testdata
c117054d groot/{rdict,rtree,rvers}: add support for T{Branch,Leaf}Object
975023e2 groot: remove old comment about write-mode of TTrees
7735669e sliceop/f64s: improve Take performances
4ca091cd all: bump x/tools@v0.1.7

Scientific Software - Peer-reviewed - Go
Published by sbinet over 4 years ago

Go-HEP - Release v0.30.0

Release v0.30.0 is out of the oven.

This release contains no major API breakage, but all around improvements and bug fixes.

fwk

updated fwk-app to use the latest gonuts/commander version that provides auto-completion of commands, flags and sub-commands.

groot

add support for TBranchObject and TLeafObject
add support for TLorentzVector (in root-dump and friends)
add support for TNtuple and TNtupleD (in root-dump and friends)

hplot

fixed a race in hplot.New where the global variable from gonum/plot was being modified without a critical section. This race appeared when one would create multiple hplot.Plot concurrently.

sliceop

improved performances of sliceop/f64s.Take. Do note that sliceop/f64s will probably be rewritten in terms of a generic implementation (and deprecated) once Go-1.18 is generally available.

AOB

That's all for today. Next cycle will probably see some work on the RNTuple front.

Changelog

baa058b5 groot/{rdict,rtree,rvers}: bump to ROOT-6.24/06
bc000321 all: bump peterh/liner, x/crypto, x/image, x/tools and ql
b57330a1 groot/{rcmd,riofs,rtree,rvers}: add rtree.Reader support for TNtuple{,D}
24ff587b all: bump klauspost/compress@v1.14.2
d1e8a6fe all: 2022 is the year of the Gopher
505ab934 fwk/cmd/fwk-app: use gonuts/commander@v0.3.x
c5711342 hplot: fix race in hplot.New on gonum/plot.DefaultFont
eaeb641a groot/rphys: test RStreamer with Bind+Unmarshaler for TLorentzVector
679c8867 groot/rphys: make Vector{2,3} implement fmt.Stringer
7851644b groot/rphys: make LorentzVector implement fmt.Stringer
07e2951c groot/rdict: leverage rbytes.{Unm,M}arshaler implementation in Bind
e370fced groot/riofs: add generation of TLorentzVector testdata
c117054d groot/{rdict,rtree,rvers}: add support for T{Branch,Leaf}Object
975023e2 groot: remove old comment about write-mode of TTrees
7735669e sliceop/f64s: improve Take performances
4ca091cd all: bump x/tools@v0.1.7

Scientific Software - Peer-reviewed - Go
Published by sbinet over 4 years ago

Go-HEP - Release v0.29.1

Release v0.29.1 is out of the oven.

This release contains no major API breakage, but all around improvements and bug fixes. Most notably, this release uses the new gonum/plot@v0.10.0 release that fixes a bunch of graphics bugs and performance improvements in plots drawing (PDF, PNG and SVG.)

arrow

We are now using a "vendored" version of github.com/apache/arrow/go without all the flight support, to cut down on the amount of dependencies this was dragging in.

groot

groot/{rdict,rtree}: handle groot types mirroring TObject inheritance tree
groot/rtree: reduce memory usage in tree-write
groot/{riofs,rtree}: handle small/big file thresholds for long-running baskets
groot/rhist: handle nil-title in YODA histograms
cmd/yoda2root: handle YODA files with a directory structure
groot/rhist: implement YODA (un)marshaler for TGraph{,Asymm}{,Errors}
groot/rhist: correctly write out TGraph{,Asymm}{,Errors}

hbook

hbook: introduce Rand1D. Rand1D represents a 1D distribution created from a hbook.H1D histogram.

hplot

hplot: add Label plotter. This plotter allows user to put labels on a plot in "normalized" or "data" coordinates.

xrootd

xrootd: fix race in file

AOB

That's all for today. Next cycle will probably see some work on the RNTuple front.

Changelog

dbb6f2e5 xrootd: fix race in file
2811be8a all: apply lint fixes
523ebcd8 ci: bump golangci-lint@v1.42
1e14bc8e all: bump go-mmap@v0.6.0
a39b0282 all: update go-mmap@v0.5.1, go-cmp@v0.5.6, xz@v0.5.10, x-tools@v0.1.6
b636438e fit: use cmpimg.CheckPlotApprox where needed
fb956a53 all: update reference files
64a270d1 hplot: migrate to new gonum/plot font cache
8c95ddff all: bump to gonum.org/v1/{gonum@v0.9,plot@v0.10}
26d966e5 ci: fix (build+module) cache of paths
685d3dfb ci: add git-config for Windows
660723ff ci: use GHA+Windows
c1c804eb ci: use Go-1.16 for AppVeyor
baf51613 all: drop use of io/ioutil
c467a582 groot/cmd/root-print: add -regen mechanism
cb8ccf5d all: bump klauspost/compress@v1.13.4
72b99d88 all: add Go-1.17, drop Go-1.15
576edb60 groot/rhist: add att{line,fill,marker} fields to graph
4622cac6 groot/rhist: correctly write out TGraph{,Asymm}{,Errors}
ceb43b51 groot/rhist: implement YODA (un)marshaler for TGraph{,Asymm}{,Errors}
a721c455 all: apply gofmt -w -s
b410620b groot: bump to pierrec/lz4@v4.1.8, klauspost/compress@v1.13.1
dcbc8102 ci: re-enable static builds where possible
4a636e4b ci: use netgo tag in mk-release
36092fc2 all: bump to ROOT-6.24.00
c2bd848c cmd/yoda2root: handle YODA files with a directory structure
b19b7758 groot/rhist: handle nil-title in YODA histograms
84ca3d76 groot/riofs: use SMHiggsToZZTo4L.root from ccxrootdgotest instead of eospublic.cern.ch
226a27a5 ci: add support for Ubuntu-20 and headless X11 mode
7024f736 hep: use light-arrow instead of apache/arrow
8aeeba84 hep: remove Travis-CI badge
eda01ec6 all: update license blurb
73dbb27c all: prepare for go:build directive
a2762f6b fastjet/internal/plot: remove old plot code
947bb632 hep: bump klauspost/compress@v1.11.8
4fe1e1ab all: bump to Go-1.16
720a7c27 doc: add link to forum discussion
36c2894d hep: bump npyio@v0.5.2
bd26c299 fmom: use func-based spatial/r3 API
160f5eb2 hep: bump klauspost/compress to v1.11.7
cc69431f all: 2021 is the year of the Gopher
8fb7154d hplot,pawgo: use new vg-gio
0410c37e all: use main instead of master
4b45b719 hplot: add Label plotter
262cb504 fmom: implement VecOf
9a1d24de fmom: add SetPtEtaPhi{E,M} to PxPyPzE
e07f1889 hbook: introduce Rand1D
2edf3b7b groot/{riofs,rtree}: handle small/big file thresholds for long-running baskets
1d1c78d1 groot/rtree: reduce memory usage in tree-write
8c71fb89 groot/riofs: fix overlaping blocks w/ big-file mode
43e74de9 all: bump versions of arrow, gokrb5, klauspost/compress, npyio and ql
f5eb1e63 ci: bump lint to v2
a6f1eec6 all: use xyzexampletest Go idiom
853cfac7 fit: remove exponential example from darwin tests
c976e7f0 all: bump gonum/plot@v0.8.1
b2e31fc2 xrootd: update for new ccxrootd test server
55f031ef groot/rdict: make sure StreamerOf panics when it must (chan,int,uint,func)
4759357b groot/{rdict,rtree}: handle groot types mirroring TObject inheritance tree

Scientific Software - Peer-reviewed - Go
Published by sbinet over 4 years ago

Go-HEP - Release v0.28.3

Scientific Software - Peer-reviewed - Go
Published by sbinet over 5 years ago

Go-HEP - Release v0.28.2

use new gonum/plot@v0.8.1 and its fix for fonts

Scientific Software - Peer-reviewed - Go
Published by sbinet over 5 years ago

Go-HEP - Release v0.28.1

update for new ccxrootd test server.

Scientific Software - Peer-reviewed - Go
Published by sbinet over 5 years ago

Go-HEP - Release v0.28.0: write structured-tree

Release v0.28.0 is out of the oven.

This release contains a major new groot related feature: the ability to write non-split user types, by way of TBranchElement and TLeafElement.

csvutil

add support for writing slices (thanks Chinmaya Krishnan Mahesh (chin123)!)

groot

add support for writing large files (i.e.: with size > 2Gb)
changed the way groot streamers are created and handled (read/write) from ROOT ones
removed rtree.Scanner and rtree.TreeScanner types, in favor of rtree.Reader
add support for reading std::bitset<N>
add initial support for reading std::map<K,V>
introduce rtree/rfunc.Formula to easily create and use "user-based" formulae (thanks Romain Madar (rmadar) for the improvements!)

hplot

use gonum/plot/vg's Gio backend for pawgo
various fixes and improvements to hplot plotters (thanks Romain Madar (rmadar)!)

AOB

That's all for today. Next cycle will probably see some work on the structured tree writing (mostly consolidation and adding missing features) and perhaps some performance improvements on the writing side of things.

ChangeLog

653c8f36 all: update Go-mod
fe7367f2 groot/rtree: test any, arr-of(any), vec(any) and vec(vec(any))
0db30afb groot/rtree: test c-var-len-arrays
de9193a3 groot/rtree: implement writing of structured-event ntuples
33c9bd26 groot/rdict: streamline StreamerOf, add support for slices
2e69fce9 groot/rtree: add support for no-split to WriteVarsFromStruct
43b8c835 groot/rmeta: add stdint to CxxBuiltins
60cf05d2 groot/rdict: add StreamerLoop visitor
2bfd292b groot/{cmd/root-ls,rcmd}: refactor root-ls as a rcmd.List
c5e9a68b groot/{,rarrow,rcmd,riofs,rtree}: properly handle slices of n-dim arrays
ee067e76 groot/{rarrow,rtree}: reduce Leaf API
cdbafa1f groot/{,rarrow,rtree}: better leaf-kind/shape/type handling
ae4a840d groot/rtree: simplify Tree/Branch/Leaf interfaces
16ed6fb7 cmd,groot: remove rtree.{Tree,}Scanner
dba77171 groot/rsql/rsqldrv: use rtree.Reader in lieu of Scanner
d6ee5e87 groot/rsrv: use rtree.Reader in lieu of Scanner
eed0f08c groot/rtree: use Reader in lieu of Scanner
a3988c10 groot/cmd/root-gen-datareader: use Reader in lieu of Scanner
1c0b336c groot/rcmd: use rtree.Reader in lieu of rtree.Scanner
cf246da1 groot/rtree: fix rchain entry offsets computation
4e85dd24 groot/rarrow: use rtree.Reader instead of rtree.Scanner
6a063617 cmd/root2npy: handle ndim arrays
0be35e71 groot/{rarrow,rcmd,rdict,riofs}: properly handle ndim-arrays
4d49a780 groot/rtree: use flattenArrayType in rleaf
30b9ba32 groot/rtree: streamline use of rtree.ReadVarsFromStruct in scanners
c0d6fad2 groot: introduce new x-flat-tree file, testing more ROOT builtins
20003c6f groot/rdict: std-bitset cosmetics
06d30c23 groot/{rcmd,rdict,riofs,rtree}: handle map
bd316477 groot/rdict: fix std::string streamer usage, add preliminary support for std::map-member-wise streamer
51ce0f59 groot/{rdict,riofs}: add support for std::map with ROOT-Cling dict
2381e6ae groot/{rbytes,rcmd,rdict,riofs,rtree}: add support for bitset
400e2b6f groot/internal/rtests: add support for rootcling dictionary generation
7536fc78 groot/{rcmd,rtree}: handle Geant4 recovery baskets
f4d329eb groot/rdict: add initial test for ObjectFrom
43282104 groot/rdict: impl r/w rmeta.AnyP, rmeta.Anyp
cabbd2cc groot/rdict: test r/w-streamer elem round-trip
012823d7 groot/rdict: streamline rmeta.ULong64+rmeta.Long64 handling
e675e8f2 groot/rdict: support finding counters across multi-hop inheritance hierarchies
664ec75b groot/rdict: streamline type generation, support n-dim arrays
631f18d4 groot/{rdict,cmd/root-gen-streamer}: handle array dims via StreamerElement.ArrayDims
edce8855 groot/{rbytes,rdict}: add ArrayDims() []int to StreamerElement
d5b19154 groot/{rcmd,rdict,rtree}: introduce R/W-streamers
19855690 groot/rbytes: extend StreamerInfo interface, introduce Binder and Counter interfaces
f50205ac groot/{rbytes,rdict}: add Encoder/Decoder interfaces for ObjectWise/MemberWise kinds
ee55be2b groot/{rbytes,rdict}: add BuildStreamers API
8bb70b75 groot/rtree: split r/w test of []intXX (extract and skip []int8)
711290aa groot/rtree: workaround for std::vector
b6db3d51 groot/rdict: add test for parsing var-len D32 array
3e8885f3 groot/{rdict,rvers}: add streamer-info for THashTable
c23f8cc9 groot/rcont: implement TRefTable
a882ddeb groot/rbytes: export r/w std::vectorstd::string
4672fa25 groot/rmeta: add TypeName2Enum
a4fbb9e4 groot/rmeta: add STLstdstring to ESTLType enum
1dd68ee6 groot: add C++ streamer for TString
78f2a495 groot/rbytes: remove test of (internal detail) cap-size in ResizeXYZ
b3f3f622 hep: bump github.com/klauspost/compress@v1.10.11
b9646023 all: migrate to Gonum-v0.8
a39f822a groot: bump to ROOT-6.22/02
5c4a32ea groot/riofs: add support for writing big files
79d17b1e ci: bump to Go-1.15.x
73346da2 xroot: update tests for new ccxrootd in2p3 config
d55d1b51 groot/{rcont,rhist,riofs}: use rtests macro to emulate C++ rootls
4b4fa012 groot: bump to ROOT-6.22/00
1a9e6f77 hplot: correctly handle negative bin content in log scale
7d2c4375 hep: bump to gokrb5@v8.4.1
3f458e3a groot/riofs: handle files with no embedded streamer for std::string
98270f3f groot/rtree: introduce Reader.Reset
abe8a2be groot/{rcmd,rtree}: handle reading of vector
095a9bc6 ci: add apt-get update
0e658332 hep: add official pkg.go.dev badge
f4c3901f xrootd/xrdio: implement correct URI parser
63dcd821 ci: bump golangci-lint@v1.28
47aa4362 pawgo: test plot H2D
d898ce9f hplot/cmd/iplot,pawgo: handle multiple plot-windows, blocking app.Main
e39ab99f all: bump klauspost/compress@v1.10.10
0f5496cf csvutil: add support for writing slices
b0b8bd0b groot: bump to go-mmap/mmap@v0.4.0
234a493a xrootd: do not segfault sending data with nil clients
aa37c691 hplot/cmd/iplot,pawgo: use vg-gio + Gio
ebbd2a74 groot/riofs: use go-mmap@v0.2.0
04360d78 hbook,hplot: introduce hbook.Count, use in hplot.BinnedErrBand
521c323f groot/rdict: handle more range-specs from streamers for D32/F16
dc20b078 groot/riofs: use go-mmap/mmap
0f13550f xrootd{,/xrdio}: do not segfault closing nil clients or nil files
1e01356f all: bump to gokrb5@v8.4.0
45f0797a groot/rtree: add support for rmeta.Long64 rstreamer
122ca822 ci: add a timeout=10m for golangci-lint
5616b92d ci: add back golangci-lint
24781f7c hplot: properly handle hstacks with bands and log-y
4cf183c8 groot/internal/rcompress: streamline handling of uncompressible data blocks
364dcc3f hep: update gonum/{gonum,plot}
3050bc0c groot/cmd/root-ls: clip branch name and title at 60 and 50 chars (resp.) for better display
69f0c8d5 groot/{riofs,rtree}: fix and test concurrent writing of trees in different ROOT files
ec2e5b95 ci: (temporarily) disable golangci-lint action
fcea6526 {cmd/root2arrow,groot/rarrow}: handle leaves with embedded std::vector
e2ffbc7c cmd/root2npy: use rtree.Reader
6c0e8ee8 groot/{rtree,rcmd}: add support for recovered baskets
4e246266 ci: make travis ci less verbose
3b702ec4 groot: implement new rfunc-gen default type name convention
384b9ef3 groot/rdict: add generation of streamer info checksum
4d1e5f08 groot/rtree: fix handling of rleaf-element with a "This" StreamerSTL
5e981213 groot/rtree: add formula test for slices
351d7f17 groot: add support for extracting copyright year from code templates
c771d0c9 groot/cmd/root-gen-rfunc: first import
e1d8871a hplot: fix HStack band display logic
a8592066 ci: remove lint scheduled/cron
508d731b groot/rtree: lift WriteVar out of Writer
9448e168 groot/{rcmd,rtree}: make Copy take a Reader instead of Tree
1b6e3cf3 groot/rtree{,/rfunc}: introduce rfunc.Formula protocol
0334332a hep: add pkg.go.dev badge
488e761e groot/{riofs,rtree}: add Join of trees, associated reader and test files
3d790e70 groot/rtree: simplify Tree interface
32ae501f groot/rtree: fix typo in Chain documentation, rename tchain into chain
f94677ef ci: extract lint-check into its own workflow
50964db8 ci: fix action/cache restore key syntax
56b78f79 csvutil/csvdriver,hbook/ntup/ntcsv: use ql@v1.1.0 with driver.ConnBeginTx implementation
56834a47 all: fix lint
98694bc7 ci: weave in golangci-lint
96b87a43 csvutil/csvdriver: staticcheck fixes
54e0051a csvutil/csvdriver: implement new database/sql/driver.XyzContext
47234700 groot/rsql/rsqldrv: apply staticcheck fixes
889375c3 groot/rsql/rsqldrv: migrate to new database/sql/driver.XyzContext interfaces

Scientific Software - Peer-reviewed - Go
Published by sbinet over 5 years ago

Go-HEP - Release v0.27.0: groot faster than ROOT

Release v0.27.0 is out of the oven.

This release contains a major groot performance improvement. As can be seen from groot-bench, groot can now read ROOT files (compressed or not) faster than (naive?) C++/ROOT 6.20/04 code (w/ TTree::Branch or TTreeReader):

name time/op ReadCMS/GoHEP/Zlib-8 19.2s ± 1% ReadCMS/ROOT-TreeBranch/Zlib-8 37.5s ± 1% ReadCMS/ROOT-TreeReader/Zlib-8 26.1s ± 3% ReadCMS/ROOT-TreeReaderMT/Zlib-8 25.6s ± 5% (ROOT::EnableImplicitMT())

This was achieved by: - re-engineering the reading code into dedicated rleafXyz, rbasket, ... types - introducing a rbasket (concurrent) pre-fetcher

There are still a few low-hanging fruits to reap performances wise (reducing memory usage, reusing rbasket buffers, reusing rbasket buffers across decompression goroutines, etc...)

cmd

GoHEP gained a new command, cmd/podio-gen, that can generate Go types according to PODIO descriptions:

``` $> podio-gen -h podio-gen generates a complete EDM from a PODIO YAML file definition.

Usage: podio-gen [OPTIONS] edm.yaml

Example:

$> podio-gen -p myedm -o out.go -r 'edm4hep::->edm,ExNamespace::->exns' edm.yaml

Options: -o string path to the output file containing the generated code (default "out.go") -p string package name for the PODIO generated types (default "podio") -r string comma-separated list of rewrite rules (e.g., 'edm4hep::->edm_') ```

This command should be useful in the connection with Key4HEP.

fit

fit gained the ability to fit multivariate functions, thanks to @JCCPort (Josh Porter):

https://godoc.org/go-hep.org/x/hep/fit#example-CurveND--Plane
https://godoc.org/go-hep.org/x/hep/fit#CurveND

fmom

fmom gained a couple of functions (under the friendly pressure and w/ contributions from @rmadar (Romain Madar)): - fmom.Dot(p1, p2 P4) P4 - fmom.Boost(p P4, vec r3.Vec) P4 - fmom.BoostOf(p P4) r3.Vec

groot

groot/rtree gained a new type rtree.Reader, that allows to more easily (and faster) read data from a tree

https://godoc.org/go-hep.org/x/hep/groot/rtree#example-Reader

```go func ExampleReader() { f, err := groot.Open("../testdata/simple.root") if err != nil { log.Fatalf("could not open ROOT file: %+v", err) } defer f.Close()

o, err := f.Get("tree")
if err != nil {
    log.Fatalf("could not retrieve ROOT tree: %+v", err)
}
t := o.(rtree.Tree)

var (
    v1 int32
    v2 float32
    v3 string

    rvars = []rtree.ReadVar{
        {Name: "one", Value: &v1},
        {Name: "two", Value: &v2},
        {Name: "three", Value: &v3},
    }
)

r, err := rtree.NewReader(t, rvars)
if err != nil {
    log.Fatalf("could not create tree reader: %+v", err)
}
defer r.Close()

err = r.Read(func(ctx rtree.RCtx) error {
    fmt.Printf("evt[%d]: %v, %v, %v\n", ctx.Entry, v1, v2, v3)
    return nil
})
if err != nil {
    log.Fatalf("could not process tree: %+v", err)
}

// Output:
// evt[0]: 1, 1.1, uno
// evt[1]: 2, 2.2, dos
// evt[2]: 3, 3.3, tres
// evt[3]: 4, 4.4, quatro

}

```

root-dump has been updated to use that new way of reading data.

Experimental support for ROOT::RNtuple was added -in a still very WIP fashion- into the groot/exp/rntup package. When ROOT7 is out (or when ROOT::RNtuple is more stable), this package is expected to graduate to groot/rntup.

The rtree.Reader type has also gained the ability to declare and evaluate user provided functions, taking a list of leaf names and the function to evaluate on the tree data:

https://godoc.org/go-hep.org/x/hep/groot/rtree#example-Reader--WithFormulaFunc
https://godoc.org/go-hep.org/x/hep/groot/rtree#example-Reader--WithFormulaFromUser

```go func ExampleReader_withFormulaFunc() { f, err := groot.Open("../testdata/simple.root") if err != nil { log.Fatalf("could not open ROOT file: %+v", err) } defer f.Close()

o, err := f.Get("tree")
if err != nil {
    log.Fatalf("could not retrieve ROOT tree: %+v", err)
}
t := o.(rtree.Tree)

var (
    data struct {
        V1 int32   `groot:"one"`
        V2 float32 `groot:"two"`
        V3 string  `groot:"three"`
    }
    rvars = rtree.ReadVarsFromStruct(&data)
)

r, err := rtree.NewReader(t, rvars)
if err != nil {
    log.Fatalf("could not create tree reader: %+v", err)
}
defer r.Close()

f64, err := r.FormulaFunc(
    []string{"one", "two", "three"},
    func(v1 int32, v2 float32, v3 string) float64 {
        return float64(v2*10) + float64(1000*v1) + float64(100*len(v3))
    },
)
if err != nil {
    log.Fatalf("could not create formula: %+v", err)
}

fstr, err := r.FormulaFunc(
    []string{"one", "two", "three"},
    func(v1 int32, v2 float32, v3 string) string {
        return fmt.Sprintf(
            "%q: %v, %q: %v, %q: %v",
            "one", v1, "two", v2, "three", v3,
        )
    },
)
if err != nil {
    log.Fatalf("could not create formula: %+v", err)
}

f1 := f64.Func().(func() float64)
f2 := fstr.Func().(func() string)

err = r.Read(func(ctx rtree.RCtx) error {
    v64 := f1()
    str := f2()
    fmt.Printf("evt[%d]: %v, %v, %v -> %g | %s\n", ctx.Entry, data.V1, data.V2, data.V3, v64, str)
    return nil
})
if err != nil {
    log.Fatalf("could not process tree: %+v", err)
}

// Output:
// evt[0]: 1, 1.1, uno -> 1311 | "one": 1, "two": 1.1, "three": uno
// evt[1]: 2, 2.2, dos -> 2322 | "one": 2, "two": 2.2, "three": dos
// evt[2]: 3, 3.3, tres -> 3433 | "one": 3, "two": 3.3, "three": tres
// evt[3]: 4, 4.4, quatro -> 4644 | "one": 4, "two": 4.4, "three": quatro

} ```

groot also gained a new command: root-split.

https://godoc.org/go-hep.org/x/hep/groot/cmd/root-split

``` $> root-split -h Usage: root-split [options] file.root

ex: $> root-split -o out.root -n 10 ./testdata/chain.flat.1.root

options: -n int number of events to split into (default 100) -o string path to output ROOT files (default "out.root") -t string input tree name to split (default "tree") -v enable verbose mode ```

root-split allows to split a given tree from a file into n files.

hbook

hbook gained a new "sub-package", hbook/ntup/ntroot that provides convenience functions to expose ROOT trees as hbook ntuples.

H1D can now be subtracted, added, added and scaled (thanks @rmadar)

https://godoc.org/go-hep.org/x/hep/hbook#AddH1D
https://godoc.org/go-hep.org/x/hep/hbook#AddScaledH1D
https://godoc.org/go-hep.org/x/hep/hbook#SubH1D

hbook/yodacnv now supports YODA format version 2 (and its YAML-based metadata description.)

hplot

Many improvement to hplot have been applied thanks again to the interesting suggestions from @rmadar (and his contributions.) Namely, a new plotter, HStack has been provided:

hstack-example hstack-band-example

A nice convenience tool to automatically generate (with pdflatex by default) LaTeX PDF plots from the gonum.org/v1/plot/vg/vgtex backend, has been also added to the new hplot concept of a Figure:

https://godoc.org/go-hep.org/x/hep/hplot#Fig
https://godoc.org/go-hep.org/x/hep/hplot#Figure
https://godoc.org/go-hep.org/x/hep/hplot/htex#Handler

Also, a new RatioPlot plotter has been added as well:

ratio-plot

lhef

lhef was improved to be able to handle version 3 of the Les Houches File format.

sliceop/f64s

sliceop/f64s is a new package to apply some "map-reduce"-esque concepts and operations to slices of float64 values.

https://godoc.org/go-hep.org/x/hep/sliceop/f64s

The Filter, Find, Map and Take functions are available.

AOB

That's all for today. Writing trees with structured data is still on the roadmark, together with (probably) some performance improvements on the writing side of things. Also, already in, is a feature similar to TTreeFriends: rtree.Join(trees []rtree.Tree).

Scientific Software - Peer-reviewed - Go
Published by sbinet about 6 years ago

Go-HEP - Release v0.26.0: rtree.Formula & hplot improvements

Scientific Software - Peer-reviewed - Go
Published by sbinet about 6 years ago

Go-HEP - Release v0.25.0: fmom.Boost, LHEF and groot.Reader

Scientific Software - Peer-reviewed - Go
Published by sbinet about 6 years ago

Go-HEP - Release v0.24.1: ROOT-6.20

drop golang.org/x/xerrors
bump to ROOT-6.20/00
introduce cmd/podio-gen a PODIO-like generator
add ability to align axes in a hplot.TiledPlot
improve compatibility w/ ROOT::TTree::{Draw,Scan}
make groot/rtree.Writer.Close() idempotent (so it can be defer'd)

Scientific Software - Peer-reviewed - Go
Published by sbinet about 6 years ago

Go-HEP - Release v0.23.0: root2fits & fits2root

Release v0.23.0 is fresh from the oven.

This release contains a couple of groot related features and bugfixes.

groot

groot gained 2 new commands:

cmd/root2fits converts a ROOT TTree into a FITS table:

``` $> root2fits -h root2fits converts the content of a ROOT tree to a FITS (binary) table.

Usage: root2fits [OPTIONS] -f input.root

Example:

$> root2fits -f ./input.root -t tree

Options: -f string path to input ROOT file name -o string path to output FITS file name (default "output.fits") -t string name of the ROOT tree to convert ```

cmd/fits2root converts a FITS table into a ROOT TTree:

``` $> fits2root -h fits2root converts the content of a FITS table to a ROOT file and tree.

Usage: fits2root [OPTIONS] -f input.fits

Example:

$> fits2root -f ./input.fits -t MyHDU

Options: -f string path to input FITS file name -o string path to output ROOT file name (default "output.root") -t string name of the FITS table to convert ```

Also, many root-xyz commands have been refactored into a simple shim executable that calls into the new groot/rcmd so users (and groot tests) can more easily customize or re-use, say, root-ls, root-dump, ... through an API instead of spawning a sub-process. This refactor is especially useful for rcmd.Merge, where users can register their own merge strategy for their own types (by implementing the root.Merger interface.)

groot/rtree can now correctly read multi-leaves branches.

And support for TLeafF16, TLeafD32, TProcessID, TRef, TRefArray, TVector2, TVector3, TLorentzVector and TFeldmanCousins has been added.

hplot

fixes for HLine and VLine have been applied: HLine/VLine outside a canvas were incorrectly drawn.

xrootd

migration to gokrb5/v8

ChangeLog

398059a groot/rbase: add initial support for tracking Refs
8c42445 groot: introduce root.UIDer interface
8d26a4a groot{,rdict,rvers}: first stab at TBits
4b94d19 groot{,rdict,rvers}: first stab at TProcessUUID
84a6e03 ci: bump appveyor to Go-1.13
6e0dddd groot{,rdict,rphys,rvers}: add TFeldmanCousins
be0efa3 groot/rphys: properly handle version-2 of TVector{2,3}
c9a6535 groot{,rdict,rphys,rvers}: add TVector2
77598d3 hplot: fix HLine/VLine for out of canvas lines
64086e9 hep: update Go modules
20a7b06 hep,xrootd/xrdproto/auth: migrate to gokrb5/v8
4900ca2 groot: implement TVector3, TLorentzVector
808cc5b groot: add TRefArray
dbf217c groot: add first stab at a TRef implementation
683c737 hbook: fix doc-example of H1D.Integral after Binning1D API migration
c1fb32e cmd/root2arrow: improve automatic installation of arrow-cat
237736a groot/{internal,rdict}: add list-groot-{sinfos,types}
12725a7 groot: generate textual representation of streamers
9ae982c groot/rdict: handling of Long64_t members in ROOT->groot streamer generation
ca69d60 groot/rdict: better handling of groot-wrapped ROOT types in ROOT->groot streamer generation
c19782e groot/rdict: add range-parsing to StreamerElement unmarshaling
499a8f4 groot/riofs: add generation of std::map-based ROOT data tree
2376dc0 groot/riofs/gendata: simplify code generation
d5830ae groot/{rdict,riofs,rtree}: improve STL-container name arguments parsing
7118537 groot/{rdict,rtree}: rename StreamerSTL.STLVectorType into STLType
8e85fb9 xroot/xrdproto/auth/krb5: migrate to gokrb5.v7
a44b1ea hep: go mod tidy
3216604 hplot: make sure embedmd is installed+required
fb7f30f hep: remove link to opencollective
64d644a groot: implement r/w TLeafF16
d17364c groot: implement r/w TLeafD32
c9904cb groot/rdict: add support for TLeaf{D32,F16} comment/range parsing
98e75b5 groot{,/riofs,/rvers}: add support for TLeafF16 and TLeafD32
5621593 groot/{cmd/root-diff,rcmd}: refactor root-diff into rcmd.Diff
4c1484a groot/rcmd: refactor Dump into a dumpCmd struct
9e53d27 groot/{cmd/root-cp,rcmd}: export root-cp as rcmd.Copy
a1734ed groot/{cmd/root-merge,rcmd}: export root-merge as rcmd.Merge
6851230 cmd/arrow2root: use rcmd.Dump
944c4a9 cmd/fits2root: use groot/rcmd.Dump in tests
e20ccab groot: export rcmd package
098e848 groot: more pervasive use of rtests.RunCxxROOT
af6ef10 groot/rtree: handle merging of Atlas flat-tuples
1df725b groot/riofs: add support for TProcessID StreamerInfo
0e1f43f groot/{rbase,rtypes}: implement TProcessID
1978a74 groot{,/rvers}: add support for TProcessID
ebbd064 groot/rdict: better error message
d3b1b41 groot/rmeta: better error message
cee0755 groot/rtree: add more Branch.setAddress tests
aad9221 groot/riofs: use rtests for generating testdata
98fe128 groot/{internal/rcmd,riofs}: add testfile w/ & w/o padding
4419621 groot/internal/rcmd: properly display branches w/ multi-leaves in root-dump
e25f2ad groot/rtree: support reading multi-leaves branches
3c58df0 cmd/root2fits: first import
f15977d cmd/fits2root: first import
3e92a08 all: 2020 is the year of the Gopher

Scientific Software - Peer-reviewed - Go
Published by sbinet over 6 years ago

Go-HEP - Release v0.22.0: root-merge and Happy New Year

Release v0.22.0 is fresh from the oven.

This release contains a couple of groot related features and bugfixes.

groot

Building on the work on tree-writing from v0.21.00, groot gained 3 new commands:

cmd/arrow2root converts an ARROW data file into a corresponding ROOT tree,

$> arrow2root testdata/primitives.file.data $> root-ls -t output.root === [./output.root] === version: 61804 TTree tree tree (entries=10) bools "bools/O" TBranch int8s "int8s/B" TBranch int16s "int16s/S" TBranch int32s "int32s/I" TBranch int64s "int64s/L" TBranch uint8s "uint8s/b" TBranch uint16s "uint16s/s" TBranch uint32s "uint32s/i" TBranch uint64s "uint64s/l" TBranch float32s "float32s/F" TBranch float64s "float64s/D" TBranch

cmd/npy2root converts a NumPy data file into a correspondig ROOT tree,

``` $> npy2root -h npy2root converts the content of a NumPy data file to a ROOT file and tree.

Usage: npy2root [OPTIONS] input.npy

The NumPy data file format is described here:

https://docs.scipy.org/doc/numpy/neps/npy-format.html

Example:

$> npyio-ls input.npy ================================================================================ file: input.npy npy-header: Header{Major:1, Minor:0, Descr:{Type:<f8, Fortran:false, Shape:[2 3]}} data = [0 1 2 3 4 5]

$> npy2root -o output.root -t mytree ./input.npy $> root-ls -t ./output.root === [./output.root] === version: 61804 TTree mytree mytree (entries=2) numpy "numpy[3]/D" TBranch

$> root-dump ./output.root

file[./output.root] key[000]: mytree;1 "mytree" (TTree) [000][numpy]: [0 1 2] [001][numpy]: [3 4 5]

Options: -o string path to output ROOT file (default "output.root") -t string name of the output ROOT tree (default "tree") ```

groot/cmd/root-merge is the beginning of an hadd-like command, merging (for now) trees and graphs.

``` $> root-merge -h Usage: root-merge [options] file1.root [file2.root [file3.root [...]]]

ex: $> root-merge -o out.root ./testdata/chain.flat.1.root ./testdata/chain.flat.2.root

options: -o string path to merged output ROOT file (default "out.root") -v enable verbose mode

$> root-merge ./testdata/chain.flat.* $> root-ls -t ./out.root === [./out.root] === version: 61804 TTree tree my tree title (entries=10) B "B/O" TBranch Str "Str/C" TBranch I8 "I8/B" TBranch I16 "I16/S" TBranch I32 "I32/I" TBranch I64 "I64/L" TBranch U8 "U8/b" TBranch U16 "U16/s" TBranch U32 "U32/i" TBranch U64 "U64/l" TBranch F32 "F32/F" TBranch F64 "F64/D" TBranch ArrBs "ArrBs[10]/O" TBranch ArrI8 "ArrI8[10]/B" TBranch ArrI16 "ArrI16[10]/S" TBranch ArrI32 "ArrI32[10]/I" TBranch ArrI64 "ArrI64[10]/L" TBranch ArrU8 "ArrU8[10]/b" TBranch ArrU16 "ArrU16[10]/s" TBranch ArrU32 "ArrU32[10]/i" TBranch ArrU64 "ArrU64[10]/l" TBranch ArrF32 "ArrF32[10]/F" TBranch ArrF64 "ArrF64[10]/D" TBranch N "N/I" TBranch SliBs "SliBs[N]/O" TBranch SliI8 "SliI8[N]/B" TBranch SliI16 "SliI16[N]/S" TBranch SliI32 "SliI32[N]/I" TBranch SliI64 "SliI64[N]/L" TBranch SliU8 "SliU8[N]/b" TBranch SliU16 "SliU16[N]/s" TBranch SliU32 "SliU32[N]/i" TBranch SliU64 "SliU64[N]/l" TBranch SliF32 "SliF32[N]/F" TBranch SliF64 "SliF64[N]/D" TBranch ```

groot/rtree also saw some activity:

fix for correctly writing []int8 and []uint8 branch data
fix for correctly writing empty strings (subtle bug when the empty string was the first or the last to be written)
fix for correctly writing trees under (deeply) nested directories
groot/rtree.Copy to easily copy trees from one place to another (possibly across files)
support for writing leaves with n-dim arrays (n>=2)

The root-print, root-cp and root-merge commands have been improved to correctly handle (arbitrarily) nested directories.

groot gained support for compressing and decompressing data via the Zstandard library. ROOT-6.20/00 should be released with this feature as well. While working on the rcompress package (that handles all compression/decompression aspects of groot), a bit of optimization work has been carried out, resulting in ~10% optimization in compression speed and a ~40% improvement in memory usage during compression.

hbook

hbook gained 2 new APIs:

hbook.H1D#FillN that fills a 1D-histogram with a slice of data and its associated weights
hbook.H12D#FillN that fills a 2D-histogram with a slice of data and its associated weights

FUSE

Support for mounting ROOT files and remote XRootD location thru FUSE has been removed from go-hep.org/x/hep. FUSE support on macOS is not open-source anymore and the provided feature thru Go-HEP was a bit flaky (at least, the tests were flaky). The xrdfuse package, xrd-fuse and root-fuse commands have been migrated to go-hep.org/x/exp.

We might consider using the 9p protocol instead of FUSE.

AOB

See you in 2020!

ChangeLog

8a851ee all: update URL of NumPy array data file specification
934bdd7 groot/internal/rcompress: optimize compression
5b1a2dd groot/internal/rcompress: fix roundtrip compression
bd257a1 groot/internal/rcompress: add compress benchmark
5801a56 hep: update go.mod
08b1f75 groot/riofs: handle riofs.Dir in riofs.FileOf
c5200f1 groot/cmd/root-merge: first import
590a5c3 groot/rtree: make Writer implement root.Merger
dfecc37 groot/rhist: implement root.Merger interface for Graph{,Errors}
28fe971 groot/root: introduce Merger interface
1437ad2 groot,xrootd: move FUSE related code to go-hep.org/x/exp
d45349f groot/internal/rcompress: streamline error strings
1005829 groot/internal/rcompress: use rtests.RunCxxROOT to run ROOT macros
54eedba groot/internal/rcompress: add support for flate.Best{Compression,Speed} to Zstd
7c7bdf3 groot/{riofs,internal/rcompress}: add support for zstd (de)compressor
1d81a65 groot/cmd/root-print: add test for '^dir'
4f47f41 groot/cmd/root-print: support nested-dirs
9db9056 groot/cmd/root-cp: add support for nested-directories
5ee79ea groot/cmd/root-cp: add support for copying trees
11df8e6 cmd/npy2root: add support for n-dim arrays
8b51e1e groot/rtree: add support for leaves with n-dim arrays (n>=2)
049273a cmd/npy2root: first import
45e282c hbook: add H2D.FillN(xs, ys, ws)
4811e2c hbook: add H1D.FillN(xs, ws)
241c5af cmd/arrow2root,groot/rarrow: introduce rarrow.NewFlatTreeWriter
64dd534 cmd/arrow2root: first import
6ecf64a groot/rtree: introduce Copy and CopyN for copying trees
c8dbaf0 groot: introduce groot/internal/rcmd
5c5a8e9 groot/rtree: test writing []int8 and [N]int8
2f7935b groot/rtree: support writing trees w/ empty strings
3a56e78 groot/rbytes: make WriteString correctly handle empty strings as first value
ebe3617 groot/{riofs,rtree}: add support for writing n-tuples under nested directories

Scientific Software - Peer-reviewed - Go
Published by sbinet over 6 years ago

Go-HEP - Release v0.21.0: writing trees -- with slices and from structs

Release v0.21.0 is fresh from the oven.

This release contains a major new groot feature: the ability to write (simple, flat) trees, including variable-length arrays (a.k.a slices):

groot

groot supports more ROOT-4 files (as created by Geant4: TH{1,2}x and TTree)
fixed compilation on 32b systems
add groot/rtree.WriterVarsFromStruct to generate a slice of rtree.WriterVars from a user-provided struct that can be then used to fill a tree
add With{LZ4,LZMA,Zlib} and WithoutCompression function to configure whether a tree should use compression (and what kind of compression, if any)
add WithBasketSize to configure the basket size of trees/branches
add auto-flushing of branches' baskets

godoc:example ``go type Data struct { I32 int32 F64 float64 Str string ArrF64 [5]float64 N int32 SliF64 []float64groot:"SliF64[N]"` // tell ROOT/C++ the leaf name and the leaf holding the count }

const ( fname = "struct.root" nevts = 5 )

f, err := groot.Create(fname) if err != nil { log.Fatalf("%+v", err) } defer f.Close()

var evt Data

tree, err := rtree.NewWriter(f, "mytree", rtree.WriteVarsFromStruct(&evt)) if err != nil { log.Fatalf("could not create tree writer: %+v", err) }

fmt.Printf("-- created tree %q:\n", tree.Name()) for i, b := range tree.Branches() { fmt.Printf("branch[%d]: name=%q, title=%q\n", i, b.Name(), b.Title()) }

for i := 0; i < nevts; i++ { evt.I32 = int32(i) evt.F64 = float64(i) evt.Str = fmt.Sprintf("evt-%0d", i) evt.ArrF64 = [5]float64{float64(i), float64(i + 1), float64(i + 2), float64(i + 3), float64(i + 4)} evt.N = int32(i) evt.SliF64 = []float64{float64(i), float64(i + 1), float64(i + 2), float64(i + 3), float64(i + 4)}[:i] _, err = tree.Write() if err != nil { log.Fatalf("could not write event %d: %+v", i, err) } } fmt.Printf("-- filled tree with %d entries\n", tree.Entries())

err = tree.Close() if err != nil { log.Fatalf("could not write tree: %+v", err) }

err = f.Close() if err != nil { log.Fatalf("could not close tree: %+v", err) }

// Output: // -- created tree "mytree": // branch[0]: name="I32", title="I32/I" // branch[1]: name="F64", title="F64/D" // branch[2]: name="Str", title="Str/C" // branch[3]: name="ArrF64", title="ArrF64[5]/D" // branch[4]: name="N", title="N/I" // branch[5]: name="SliF64", title="SliF64[N]/D" // -- filled tree with 5 entries // -- read back ROOT file // entry[0]: {I32:0 F64:0 Str:evt-0 ArrF64:[0 1 2 3 4] N:0 SliF64:[]} // entry[1]: {I32:1 F64:1 Str:evt-1 ArrF64:[1 2 3 4 5] N:1 SliF64:[1]} // entry[2]: {I32:2 F64:2 Str:evt-2 ArrF64:[2 3 4 5 6] N:2 SliF64:[2 3]} // entry[3]: {I32:3 F64:3 Str:evt-3 ArrF64:[3 4 5 6 7] N:3 SliF64:[3 4 5]} // entry[4]: {I32:4 F64:4 Str:evt-4 ArrF64:[4 5 6 7 8] N:4 SliF64:[4 5 6 7]} ```

hplot

hplot.Function has been copied from Gonum/plot with extra support for log-y axes (automatically discarding intervals where the returned value is invalid on log-y axes)

godoc:example ```go quad := hplot.NewFunction(func(x float64) float64 { return x * x }) quad.Color = color.RGBA{B: 255, A: 255}

fun := hplot.NewFunction(func(x float64) float64 { switch { case x < 6: return 20 case 6 <= x && x < 7: return 0 case 7 <= x && x < 7.5: return 30 case 7.5 <= x && x < 9: return 0 case 9 <= x: return 40 } return 50 }) fun.LogY = true fun.Color = color.RGBA{R: 255, A: 255}

p := hplot.New() p.Title.Text = "Functions - Log-Y scale" p.X.Label.Text = "X" p.Y.Label.Text = "Y"

p.Y.Scale = plot.LogScale{} p.Y.Tick.Marker = plot.LogTicks{}

p.Add(fun) p.Add(quad) p.Add(hplot.NewGrid()) p.Legend.Add("x^2", quad) p.Legend.Add("fct", fun) p.Legend.ThumbnailWidth = 0.5 * vg.Inch

p.X.Min = 5 p.X.Max = 10 p.Y.Min = 10 p.Y.Max = 100

err := p.Save(200, 200, "testdata/functions_logy.png") if err != nil { log.Panic(err) } ``` functions-logy-example

clean-up

The whole GoHEP tree has been cleaned up to remove the use of github.com/pkg/errors and use instead x/xerrors. x/xerrors will be phased out (in favor of fmt from the stdlib) when Go-1.13 will be the oldest supported release.

This is to gain nice error reports as documented in the Go blog:

https://blog.golang.org/go1.13-errors

AOB

The GoHEP website has also gained a new /dist section, where binaries for Darwin, Freebsd, Linux and Windows are uploaded (for every new tagged release).

We'll try to release v0.22.0 with the ability to write structs and nested trees.

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

ChangeLog

a3bdc48 groot/rhist: add test for ROOT-4 H1D
6c8c87a groot: add support for ROOT-4 histograms
c5fa1b7 groot/rtree: support G4-like tree
508bea9 groot/rtree: TBranch-v6
2afe0e7 groot/rtree: TBranchElement-v1
96ca4b3 groot/rtree: TTree-v5
d1049cd groot/rtree: TBranch-v10
f3ad4a5 hplot: add support for log-y based functions
0b630d8 ci: add x-compile release script
0fa7206 ci: display C++ ROOT version
c79f47a groot/{riofs,rtree}: fix 32b build
b2bb9f0 ci: test 32b builds
5f73852 groot/internal/rtests: add ROOTError wrap test
ac4885b ci: add binary ROOT installation scripts
91c9502 groot/rtree: add WriterVarsFromStruct helper function
7379a85 groot/rtree: make NewScanVars support Slices and Arrays
9be2300 groot/rtree: test WithXXX API and ability to correctly read back trees from C++ ROOT
a1a9024 groot/internal/rtests: introduce RunCxxROOT to (more easily) run ROOT macros
ac1dc83 groot/{riofs,internal/rcompress,rtree}: introduce rcompress.Settings
f35da3b groot/{riofs,rtree}: add WithXYZ API to configure how trees should be created
813c909 ci: add mk-release script to push versions to index.golang
d12963d fwk: apply golangci suggestions
295baa5 fwk: simplify fowk.MsgStream interface
86f5de3 all: use x/xerrors
f9cbd76 groot/rtree: improve diagnostic+doc for rtree.New(Tree)Scanner
c4a04e9 groot/rarrow: fix Arrow-struct creation data-offset
b8a39b7 groot/rtree: handle auto-flush
b545af5 groot/rtree: add support var-len arrays to trees

Scientific Software - Peer-reviewed - Go
Published by sbinet over 6 years ago

Go-HEP - Release v0.20.0: writing trees

Release v0.20.0 is fresh from the oven.

This release contains a major new groot feature: the ability to write (simple, flat) trees:

groot/rtree#NewWriter

and a couple of new features for hplot.

groot

groot support has been bumped to ROOT-6.18/04
groot can now create nested directories that are compatible with ROOT. Previously, groot would create nested directories which could only be read back by groot (and uproot).

```go w, err := groot.Create("subdirs.root") if err != nil { log.Fatal(err) } defer w.Close()

dir1, err := w.Mkdir("dir1") if err != nil { log.Fatal(err) }

dir11, err := dir1.Mkdir("dir11") if err != nil { log.Fatal(err) }

err = dir11.Put("obj1", rbase.NewObjString("data-obj1")) if err != nil { log.Fatal(err) }

dir2, err := w.Mkdir("dir2") if err != nil { log.Fatal(err) }

err = dir2.Put("obj2", rbase.NewObjString("data-obj2")) if err != nil { log.Fatal(err) }

err = w.Close() if err != nil { log.Fatal(err) }

// Output: // >> subdirs.root // >> subdirs.root/dir1 // >> subdirs.root/dir1/dir11 // >> subdirs.root/dir1/dir11/obj1 // >> subdirs.root/dir2 // >> subdirs.root/dir2/obj2 ```

cmd/root2csv can now handle TLeafElement leaves of builtins
groot/riofs.File.SegmentMap has been added to riofs.File to inspect the list of ROOT records in a file
support for writing simple flat trees of builtins and arrays of builtins (no slices, yet) has been added:

godoc:example ```go type Data struct { I32 int32 F64 float64 Str string ArrF64 [5]float64 } const ( fname = "groot-flat-ntuple.root" nevts = 5 )

f, err := groot.Create(fname) if err != nil { log.Fatalf("%+v", err) } defer f.Close()

var evt Data

wvars := []rtree.WriteVar{ {Name: "I32", Value: &evt.I32}, {Name: "F64", Value: &evt.F64}, {Name: "Str", Value: &evt.Str}, {Name: "ArrF64", Value: &evt.ArrF64}, } tree, err := rtree.NewWriter(f, "mytree", wvars) if err != nil { log.Fatalf("could not create tree writer: %+v", err) }

fmt.Printf("-- created tree %q:\n", tree.Name()) for i, b := range tree.Branches() { fmt.Printf("branch[%d]: name=%q, title=%q\n", i, b.Name(), b.Title()) }

for i := 0; i < nevts; i++ { evt.I32 = int32(i) evt.F64 = float64(i) evt.Str = fmt.Sprintf("evt-%0d", i) evt.ArrF64 = [5]float64{float64(i), float64(i + 1), float64(i + 2), float64(i + 3), float64(i + 4)} _, err = tree.Write() if err != nil { log.Fatalf("could not write event %d: %+v", i, err) } } fmt.Printf("-- filled tree with %d entries\n", tree.Entries())

err = tree.Close() if err != nil { log.Fatalf("could not write tree: %+v", err) }

err = f.Close() if err != nil { log.Fatalf("could not close tree: %+v", err) }

// Output: // -- created tree "mytree": // branch[0]: name="I32", title="I32/I" // branch[1]: name="F64", title="F64/D" // branch[2]: name="Str", title="Str/C" // branch[3]: name="ArrF64", title="ArrF64[5]/D" // -- filled tree with 5 entries // -- read back ROOT file // entry[0]: {I32:0 F64:0 Str:evt-0 ArrF64:[0 1 2 3 4]} // entry[1]: {I32:1 F64:1 Str:evt-1 ArrF64:[1 2 3 4 5]} // entry[2]: {I32:2 F64:2 Str:evt-2 ArrF64:[2 3 4 5 6]} // entry[3]: {I32:3 F64:3 Str:evt-3 ArrF64:[3 4 5 6 7]} // entry[4]: {I32:4 F64:4 Str:evt-4 ArrF64:[4 5 6 7 8]} ```

hplot

hplot.VLine has been added to easily create a vertical line on a plot
hplot.HLine has been added to easily create a horitzontal line on a plot
hplot.NewBand has been added to easily create a band between two sets of data points on a plot

vline-example hline-example band-example

AOB

We'll try to release v0.21.0 shortly with the ability to write slices to a tree and perhaps structs.

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Scientific Software - Peer-reviewed - Go
Published by sbinet over 6 years ago

Go-HEP - Release v0.19.0

Release v0.19.0 is fresh from the oven.

This release contains 3 new groot-related packages:

groot/rsql/rsqldrv: a package to present a ROOT File+Tree as a SQL database, by way of implementing the database/sql/driver interface for groot
groot/rsql: a convenience package to scan (a la TTree::Scan and TTree::Draw) rtree.Tree and create hbook.H1D or hbook.H2D
groot/rarrow: a package to present rtree.Tree as Apache Arrow's Tables and Records. This will be very useful for interoperability with DataScience and/or Machine Learning toolkits.

groot

Providing groot/rarrow made a new command possible - cmd/root2arrow - that can convert a ROOT TTree stored in a ROOT file, into a sequence of Arrow records, stored into an Arrow file (or Arrow stream.)

``` $> go doc go-hep.org/x/hep/cmd/root2arrow root2arrow converts the content of a ROOT TTree to an ARROW file.

Usage of root2arrow:
  -o string
        path to output ARROW file name (default "output.data")
  -stream
        enable ARROW stream (default is to create an ARROW file)
  -t string
        name of the tree to convert (default "tree")

$> root2arrow -o foo.data -t tree ../../groot/testdata/simple.root
$> arrow-ls ./foo.data
version: V4
schema:
  fields: 3
    - one: type=int32
    - two: type=float32
    - three: type=utf8
records: 1
$> arrow-cat ./foo.data
version: V4
record 1/1...
  col[0] "one": [1 2 3 4]
  col[1] "two": [1.1 2.2 3.3 4.4]
  col[2] "three": ["uno" "dos" "tres" "quatro"]

```

Here is an example of rsql.Scan:

```go func main() { f, err := groot.Open("../testdata/simple.root") if err != nil { log.Fatal(err) } defer f.Close()

o, err := f.Get("tree")
if err != nil {
    log.Fatal(err)
}

tree := o.(rtree.Tree)

var (
    v1s []int32
    v2s []float64
    v3s []string
)

err = rsql.Scan(tree, "SELECT (one, two, three) FROM tree", func(x int32, y float64, z string) error {
    v1s = append(v1s, x)
    v2s = append(v2s, y)
    v3s = append(v3s, z)
    return nil
})

if err != nil {
    log.Fatal(err)
}

fmt.Printf("tree[%q]: %v\n", "one", v1s)   // tree["one"]: [1 2 3 4]
fmt.Printf("tree[%q]: %v\n", "two", v2s)   // tree["two"]: [1.1 2.2 3.3 4.4]
fmt.Printf("tree[%q]: %q\n", "three", v3s) // tree["three"]: ["uno" "dos" "tres" "quatro"]

} ```

And one example of rsql.ScanH1D:

```go func main() { f, err := groot.Open("../testdata/simple.root") if err != nil { log.Fatal(err) } defer f.Close()

o, err := f.Get("tree")
if err != nil {
    log.Fatal(err)
}

tree := o.(rtree.Tree)

h, err := rsql.ScanH1D(tree, "SELECT two FROM tree", nil)
if err != nil {
    log.Fatal(err)
}

fmt.Printf("entries: %v\n", h.Entries()) // entries: 4
fmt.Printf("x-mean: %v\n", h.XMean())    // x-mean: 2.75

} ```

AOB

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Scientific Software - Peer-reviewed - Go
Published by sbinet about 7 years ago

Go-HEP - Release v0.18.0

Release v0.18.0 is fresh-ish from the oven.

This release contains the beginning of an XRootD server in Go (thanks Mikhail!) as well as yet more groot improvements.

fit

fit: update for new gonum.org/v1/gonum/optimize API
fit: use new gonum/{diff/fd,optimize} API

groot

groot/riofs.Open and thus groot.Open have been modified to only open local files by default. A "plugin" mechanism has been devised to be able to optionally install plugins to deal with, e.g., HTTP-served and XRootD-served files. To enable these plugins, one just need to "blank import" the following packages:

go import ( _ "go-hep.org/x/hep/groot/riofs/plugin/http" _ "go-hep.org/x/hep/groot/riofs/plugin/xrootd" )

The groot/cmd/root-xxx commands have been modified to automatically import the above packages. Users are expected to do the same if they want to transparently be able to read such files. The rationale being that groot/riofs should by default try to always create static binaries: to read the HTTP- and XRootD-served files one needs to import net/http that require some amount of dynamic linking.

The groot package also gained a new command:

``` $> root-gen-type -h Usage: root-gen-type [options] input.root

ex: $> root-gen-type -p mypkg -t MyType -o streamers_gen.go ./input.root

options: -o string output file name -p string package import path -t string comma-separated list of (regexp) type names (default ".*") -v enable verbose mode ```

root-gen-type automatically creates the streamer code from an input ROOT file (not unlike the old genreflex or the new rootcling) as well as a Go struct compatible with the data contained in the (C++, on-disk) streamers (kind of like the C++ TTree::MakeClass.)

groot: add support for TMap
groot/rdict: add support for StreamerObject{,Pointer}
groot/cmd/root-ls: lazily load data off disk
groot/rvers: migrate to ROOT-6.16/00
groot: add support for TClonesArray
groot: introduce ROOT open plugin mechanism
groot/cmd/root-gen-type: first import
groot: support TArrayC, TArrayS and TArrayL
groot/rdict: add support for slices of Object-s or ObjectAny-s
groot: add initial support for Float16t and Double32t in root-gen-type

hbook

hbook/ntup: add an example on how to create histos from CSV
hbook/ntup: add examples for Scan and ScanH1D methods

hep

hep: add Version to API

hplot

hplot: add support for H1D-logy

xrootd

xrootd: add ping, mkdir, rm, and rmdir support to the server
xrootd/cmd/xrd-ls: fix output format
xrootd/xrdproto/query: update example for new XRootD version

AOB

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Scientific Software - Peer-reviewed - Go
Published by sbinet about 7 years ago

Go-HEP - Release v0.17.1

Release v0.17.1 is fresh-ish from the oven.

This new release continues the refactoring and consolidation work of the groot package (meant to replace rootio.)

brio

brio/cmd/brio-gen gained some documentation

groot

groot/rdict has seen a bunch of work to create user-type StreamerInfos,
groot/cmd/root-gen-streamer: new command to automatically generate a StreamerInfo and its StreamerElements given a ROOT or user type,
groot/rdict now properly handles streamers with arrays of builtins, and properly visits std::vector<T> fields (where T is a builtin)
groot/cmd/root-dump: now properly handle root.List values
groot dropped the use of gofrs/uuid and replaced it with hashicorp/go-uuid. gofrs/uuid dropped support for Go modules and broke the build of Go-HEP.

Here is usage example of the new root-gen-streamer command:

```sh $> root-gen-streamer -help Usage: root-gen-streamer [options]

ex: $> root-gen-streamer -p image -t Point -o streamersgen.go $> root-gen-streamer -p go-hep.org/x/hep/hbook -t Dist0D,Dist1D,Dist2D -o foostreamer_gen.go

options: -o string output file name -p string package import path -t string comma-separated list of type names -v enable verbose mode

```

xrootd

The xrootd/client and xrootd/server packages have been merged into a single package, xrootd. This simplified the import structure as well as reduced the amount of boilerplate code that was duplicated between the two packages.

For users of xrootd/client:

```diff diff --git a/xrootd/cmd/xrd-ls/main.go b/xrootd/cmd/xrd-ls/main.go index 784a5c7..9961802 100644 --- a/xrootd/cmd/xrd-ls/main.go +++ b/xrootd/cmd/xrd-ls/main.go @@ -31,7 +31,7 @@ import ( "text/tabwriter"

    "github.com/pkg/errors"

xrdclient "go-hep.org/x/hep/xrootd/client"

  "go-hep.org/x/hep/xrootd"
"go-hep.org/x/hep/xrootd/xrdfs"
"go-hep.org/x/hep/xrootd/xrdio"

) @@ -93,7 +93,7 @@ func xrdls(name string, long, recursive bool) error {

ctx := context.Background()

  c, err := xrdclient.NewClient(ctx, url.Addr, url.User)

  c, err := xrootd.NewClient(ctx, url.Addr, url.User)
if err != nil {
        return errors.Errorf("could not create client: %v", err)
}

```

For users of xrootd/server:

```diff diff --git a/xrootd/cmd/xrd-srv/main.go b/xrootd/cmd/xrd-srv/main.go index 2a09dbd..230bf31 100644 --- a/xrootd/cmd/xrd-srv/main.go +++ b/xrootd/cmd/xrd-srv/main.go @@ -14,7 +14,7 @@ import ( "os" "os/signal"

"go-hep.org/x/hep/xrootd/server"
"go-hep.org/x/hep/xrootd" )

func init() { @@ -56,7 +56,7 @@ func main() { log.Fatalf("could not listen on %q: %v", *addr, err) }

srv := server.New(server.NewFSHandler(baseDir), func(err error) {
srv := xrootd.NewServer(xrootd.NewFSHandler(baseDir), func(err error) { log.Printf("an error occured: %v", err) }) ```

AOB

Support for writing TTrees didn't make it under the X-mas tree. This has been converted to a NYE resolution, though.

Before tackling this big item, support for reading TClonesArray is on the way (and tracked in the sbinet/hep#issue-419 branch.)

A proposal for data frames is in the works in the gonum/exp repository. Feel free to comment on the associated pull request (#19) or on the gonum-dev forum.

Finally, this is GSoC proposal season. I (@sbinet) will probably send a GDML oriented proposal. Feel free to send or discuss yours on the go-hep mailing list.

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Scientific Software - Peer-reviewed - Go
Published by sbinet over 7 years ago

Go-HEP - Release v0.16.0

Release v0.16.0 is fresh from the oven.

This new release introduces a new package, groot, that replaces rootio.

groot & rootio

groot

groot is a new package that is meant to replace rootio. For backward compatibility, rootio will remain in the repository for a couple of releases (presumably until v1.0.0.)

groot is the result of refactoring rootio in a couple of more focused packages with clear API boundaries:

groot/rbase: definitions of basic ROOT classes (Object, Named, ObjString, ...)
groot/rbytes: definitions of types useful for serializing and deserializing ROOT data buffers, interfaces to interact with ROOT's metadata classes such as StreamerInfo and StreamerElements.
groot/rcont: definitions of ROOT container types (TList, THashList, TObjArray, TArrayX)
groot/rdict: definitions of ROOT streamers (TStreamerArtificial, TStreamerLoop, ...)
groot/rhist: definitions of ROOT types related to histograms and graphs (TH1x, TH2x, TGraph, TGraphErrors, TGraphAsymmErrors)
groot/riofs: low-level types and functions to deal with opening and creating ROOT files; users should prefer using the groot package to open and create ROOT files
groot/root: ROOT core interfaces (Object, Named, ObjArray, ...)
groot/rsrv: exposes HTTP end-point to manipulate ROOT files, plot and create histograms and graphs from files or trees
groot/rtree: interface to decode, read, concatenate and iterate over ROOT Trees
groot/rtypes: rtypes contains the means to register types (ROOT ones and user defined ones) with the ROOT type factory system
groot/rvers: rvers contains the ROOT version and the classes' versions groot is supporting and currently reading.

Interacting with ROOT files should be performed with the groot package:

```go import "go-hep.org/x/hep/groot"

func F() { f1, err := groot.Open("some/file.root") f2, err := groot.Create("some/other.root") } ```

groot has the needed bootstrap code that allows to automatically import the registration code for all the ROOT types groot knows how to handle.

Here is a quick and dirty Rosetta code for migrating to groot:

rootio.H1 → groot/rhist.H1
rootio.H2 → groot/rhist.H2
rootio.Graph → groot/rhist.Graph
rootio.Tree → groot/rtree.Tree
rootio.ChainOf → groot/rtree.ChainOf
rootio.Scanner → groot/rtree.Scanner
rootio.File → groot/riofs.File
rootio.Directory → groot/riofs.Directory
rootio.Open → groot.Open
rootio.Create → groot.Create

groot/rsrv & root-srv

root-srv has been refactored to extract the pure plot/file interaction machinery from the GUI part. The plot creation and the ROOT file interaction parts have been refactored into a new package go-hep.org/x/hep/groot/rsrv that contains a couple of HTTP end-points that can be reused in third-party packages or applications.

rsrv exposes a REST API that expects JSON requests (OpenFileRequest, PlotH1Request, PlotTreeRequest, ...) and returns JSON responses. The HTTP end-points are attached to the rsrv.Server type.

root-srv can now open ROOT files served over xrootd.

hbook

hbook now exposes a Bin method on H{1,2}D to retrieve a bin by its (x,y) coordinates.

hplot

Following an issue raised on gonum/plot, hplot now creates histograms with transparent background by default.

rio

Improved test coverage (by adding some more tests.)

xrootd

The low-level bits for the following requests have been implemented:

kXR_query
kXR_prepare
kXR_endsess
kXR_locate
kXR_decrypt
kXR_admin

The first steps to support the "host" security provider have also been implemented.

We are still missing the implementation for the GSI authentification protocol: still waiting on xrootd to provide specifications for this protocol (progress is tracked here: issue-757.)

Improved test coverage.

AOB

We will try to have preliminary support for writing TTrees in the next release. That should be fun.

Interoperability with Apache Arrow Arrays is in the works. It might even prove to be easier to support Apache Arrow first and then implement TTrees writing support on top of that. We will see...

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Scientific Software - Peer-reviewed - Go
Published by sbinet over 7 years ago

Go-HEP - Release v0.15.0

Release v0.15.0 is fresh from the oven.

This new release dropped explicit support for Go-1.8.x but, in turn, gained support for Go-1.11.x and "Go Modules". In a nutshell, Go modules allow to explicitly declare what are the dependencies a given module needs for the go build tool to successfully build it. And, more importantly, Go modules allow to explicitly declare what are the needed versions of these dependencies, essentially making a build completely reproducible.

You can find more informations about Go modules over there:

https://research.swtch.com/vgo
https://github.com/golang/go/wiki/Modules

Currently, modules are only tested in Travis-CI, on the Go master branch. But as Go-1.12.x will get closer and modules get more ubiquitous, we'll gradually switch to "Go modules" being the mainstream way to build Go-HEP.

Do not hesitate to report any issues you encounter when building with GO111MODULE=on enabled.

rootio

Another big news for the v0.15.0 release is the support for writing ROOT files:

writing TObjStrings, TH1x, TH2x, TGraph, TGraph{,Assymm}Errors is in,
support for writing compressed ROOT files as well (including lz4, lzma and zlib)
2 new ROOT-related commands:
- cmd/yoda2root: a command to convert YODA files into ROOT ones (so: histograms and scatters)
- rootio/cmd/root-cp: a command to extract objects from a ROOT file into a new ROOT file

To support writing TH1x, TH2x and TGraphs, hbook types have been modified to export most of their fields -- so one can create a rootio.H1D from a hbook.H1D. This enabled hbook/rootcnv to gain 3 new functions:

rootcnv.FromH1D: a function that converts an hbook.H1D into a rootio.H1D, loosing a bit of informations along the way (ROOT isn't as precise as hbook or YODA are)
rootcnv.FromH2D: a function that converts hbook.H2Ds into rootio.H2Ds,
rootcnv.FromS2D: a function that converts hbook.S2D into rootio.TGraphAsymmErrors.

rootio & xrootd

Finally, we have received 2 patches from Paul Seyfert (a.k.a pseyfert), our first "CERNois" committer :). Paul enhanced the UI of root-ls and xrd-ls to better deal with nested directories (and how they are displayed) in both of these commands. Thanks Paul!

Examples

Without further ado, here is how you would create a ROOT file, with lz4 compression, containing a TObjString:

```go func main() { w, err := rootio.Create("out.root", rootio.WithLZ4(flate.BestCompression)) if err != nil { log.Fatal(err) } defer w.Close()

var (
    k = "my-objstring"
    v = rootio.NewObjString("Hello World from Go-HEP!")
)

err = w.Put(k, v)
if err != nil {
    log.Fatal(err)
}

fmt.Printf("wkeys: %d\n", len(w.Keys()))

err = w.Close()
if err != nil {
    log.Fatalf("could not close file: %v", err)
}

} ```

and here is how you would use root-cp:

``` $> root-cp -h Usage: root-cp [options] file1.root [file2.root [...]] out.root

ex: $> root-cp f.root out.root $> root-cp f1.root f2.root f3.root out.root $> root-cp f1.root:hist* f2.root:h2 out.root

options:

$> root-cp ./testdata/graphs.root:g* out.root $> root-cp root://xrootd.example.org/file.root:hist.* out.root ```

More ROOT files writing examples can be found here:

AOB

We will try to have preliminary support for writing TTrees in the next release. That should be fun.

Interoperability with Apache Arrow Arrays is still on the table. It might even prove to be easier to support Apache Arrow first and then implement TTrees writing support on top of that. We will see...

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Scientific Software - Peer-reviewed - Go
Published by sbinet over 7 years ago

Go-HEP - Release v0.14.0

Release v0.14.0 is fresh from the oven.

This release is the result of some massive work in the xrootd package thanks to Mikhail Ivchenko (a.k.a @EgorMatirov), our Google Summer of Code 2018 student.

While GSoC-2018 is now over, it's time to reflect on what Mikhail wrote:

an almost complete xrootd client, compatible with the C++ implementation;
xrd-fuse, a command to mount the contents of a remote XRootD server, locally;
the beginnings of an xrootd server.

Most notably, the client package allowed to:

to create the xrd-cp and xrd-ls commands that copy and list the contents of a remote XRootD server,
to seamlessly read ROOT files over XRootD.

The client package handles authentication with the unix and kerberos protocols. Unfortunately, authentication via GSI couldn't be implemented because there were no publicly available specifications for that protocol, see xrootd/xrootd#757 for more details.

Here is the final report of this year's GSoC:

https://gist.github.com/EgorMatirov/303e4019606677d1838677a5a7145420

Thanks a lot Mikhail, hope we'll see you around :)

Another big new feature is the ability to write ROOT files, directly with go-hep/rootio. This is still very much a work in progress, though, as only writing "empty" ROOT files or writing ROOT files with TObjStrings have been explicitly tested. Next release should see explicit support for writing histograms and graphs.

Lastly, improvements on the build and continuous integration procedure have been applied during this release cycle:

added AppVeyor CI, for Windows:
added code coverage, with codecov.io:

Code coverage improvements

The following packages have been updated, with additional tests, to improve their code coverage:

brio, csvutil, csvutil/csvdriver,
fit, fmom,
heppdt, hepmc, hepevt,
hbook/ntup, hplot,
lcio, lhef,
rootio, sio,
xrootd.

Still some more work is needed to bring code coverage to a saner level (from ~55% to ~70-80%.) Help more than welcome: it's "just" a matter of creating examples and tests.

hepevt

AsciiEncoder was renamed into simply Encoder
AsciiDecoder was renamed into simply Decoder

rootio

As noted above, it is now possible to create ROOT files. The rootio package has a couple of examples:

Here is how you would create a ROOT file with one TObjString in it:

```go package main

import ( "fmt" "log"

"go-hep.org/x/hep/rootio"

)

func main() { w, err := rootio.Create(fname) if err != nil { log.Fatal(err) } defer w.Close()

var (
    k   = "my-objstring"
    v   = rootio.NewObjString("Hello World from Go-HEP!")
)

err = w.Put(k, v)
if err != nil {
    log.Fatal(err)
}

fmt.Printf("wkeys: %d\n", len(w.Keys()))

err = w.Close()
if err != nil {
    log.Fatalf("could not close file: %v", err)
}

} ```

It is quite possible the API may change as we gain experience with what it ought to look like in a Go world. e.g. it is possible that rootio.Directory.Put would get clever enough to automatically translate a Go builtin, like string, in the ROOT equivalent, say TObjString, on the fly. Still pondering on that...

Again, still a lot of work to do on the writing side of things:

support for TH1x,
support for TH2x,
support for TGraph, TGraphErrors, TGraphAsymErrors
support for TTree (this will take some time)
support for user-provided types.

xrootd

The xrootd/server package ships with support for the following XRootD requests:

open, read, close,
write, stat, truncate, sync, rename.

There is no authentication support (yet) on the server: DO NOT RUN THIS ON PUBLICLY ACCESSIBLE MACHINES :)

The xrootd package gained a new sub-command:

xrootd/cmd/xrd-srv

``` $> xrd-srv -h xrd-srv serves data from a local filesystem over the XRootD protocol.

Usage:

$> xrd-srv [OPTIONS]

Example:

$> xrd-srv /tmp $> xrd-srv -addr=0.0.0.0:1094 /tmp

Options: -addr string listen to the provided address (default "0.0.0.0:1094") ```

AOB

Support for Go-1.6 and Go-1.7 has been dropped. Please upgrade to the latest and finest Go version (1.11 is around the corner, with support for Go modules.)

Another interesting possible development avenue: exposing ROOT TTrees as Apache Arrow Arrays. This would allow for a better interoperability with that ecosystem and the tools it provides for data science and data analysis in general. Help wanted!

Scientific Software - Peer-reviewed - Go
Published by sbinet almost 8 years ago

Go-HEP - Release v0.13.0

Release v0.13.0 is fresh from the oven.

This release ships with major improvements in the xrootd implementation and a few fixes in rootio.

rootio

leveraging the work that happened in xrootd, rootio is now able to read files over [x]root:

```go import ( "go-hep.org/x/hep/rootio" )

func foo() { f, err := rootio.Open("root://ccxrootdgotest.in2p3.fr:9001/tmp/rootio/testdata/small-flat-tree.root") if err != nil { ... } defer f.Close() } ```

all the root-xyz commands can now also leverage xrootd: $> root-ls -t root://ccxrootdgotest.in2p3.fr:9001/tmp/rootio/testdata/small-flat-tree.root === [root://ccxrootdgotest.in2p3.fr:9001/tmp/rootio/testdata/small-flat-tree.root] === version: 60806 TTree tree my tree title (entries=100) │ Int32 "Int32/I" TBranch │ Int64 "Int64/L" TBranch │ UInt32 "UInt32/i" TBranch │ UInt64 "UInt64/l" TBranch │ Float32 "Float32/F" TBranch │ Float64 "Float64/D" TBranch │ Str "Str/C" TBranch │ ArrayInt32 "ArrayInt32[10]/I" TBranch │ ArrayInt64 "ArrayInt64[10]/L" TBranch │ ArrayUInt32 "ArrayInt32[10]/i" TBranch │ ArrayUInt64 "ArrayInt64[10]/l" TBranch │ ArrayFloat32 "ArrayFloat32[10]/F" TBranch │ ArrayFloat64 "ArrayFloat64[10]/D" TBranch │ N "N/I" TBranch │ SliceInt32 "SliceInt32[N]/I" TBranch │ SliceInt64 "SliceInt64[N]/L" TBranch │ SliceUInt32 "SliceInt32[N]/i" TBranch │ SliceUInt64 "SliceInt64[N]/l" TBranch │ SliceFloat32 "SliceFloat32[N]/F" TBranch │ SliceFloat64 "SliceFloat64[N]/D" TBranch
support for seeking (i.e.: event random access) has been added to scanners connected to chains of rootio.Trees,
rootio can now automatically generate streamers for std::vector<T> when a streamer for T exists,
rootio has been updated to v2 of pierrec/lz4 library to decode LZ4 compressed ROOT files.

xrootd

support for ping and protocol requests
support for dirlist, open, close and sync requests
support for read and write requests
support for rm, rmdir and truncate requests
support for stat, vstat, statx, mkdir, mv and chmod requests
support for signing requests
support for auth+unix request
introduction of the xrd-cp command to copy files from a remote xrootd server:

``` $> go doc go-hep.org/x/hep/xrootd/cmd/xrd-cp Command xrd-cp copies files and directories from a remote xrootd server to local storage.

Usage:

$> xrd-cp [OPTIONS] <src-1> [<src-2> [...]] <dst>

Example:

$> xrd-cp root://server.example.com/some/file1.txt .
$> xrd-cp root://gopher@server.example.com/some/file1.txt .
$> xrd-cp root://server.example.com/some/file1.txt foo.txt
$> xrd-cp root://server.example.com/some/file1.txt - > foo.txt
$> xrd-cp -r root://server.example.com/some/dir .
$> xrd-cp -r root://server.example.com/some/dir outdir

Options:

-r  copy directories recursively
-v  enable verbose mode

```

introduction of the xrd-ls command to list the contents of directories on a remote xrootd server:

``` $> go doc go-hep.org/x/hep/xrootd/cmd/xrd-ls Command xrd-ls lists directory contents on a remote xrootd server.

Usage:

$> xrd-ls [OPTIONS] <dir-1> [<dir-2> [...]]

Example:

$> xrd-ls root://server.example.com/some/dir
$> xrd-ls -l root://server.example.com/some/dir
$> xrd-ls -R root://server.example.com/some/dir
$> xrd-ls -l -R root://server.example.com/some/dir

Options:

-R  list subdirectories recursively
-l  use a long listing format

```

a convenience xrootd/xrdfs.FileSystem interface has been introduced to model interacting with the remote xrootd server's filesystem, following the os package API
a convenience xrootd/xrdfs.File interface has been introduced to model interacting with the remote xrootd file, following the os.File API
a convenience xrootd/xrdio.File type, implementing various io.Xyz interfaces has been introduced as well.

Scientific Software - Peer-reviewed - Go
Published by sbinet almost 8 years ago

Go-HEP - Release v0.12.0

Release v0.12.0 is fresh from the oven.

This release is first one to introduce preliminary support for vgo, the official Go way to handle versioning. vgo is still in flux: the first Go version with experimental opt-in support should be Go 1.11 (to be released in August 2018.) Obviously, on the Go-HEP side, adjustments will probably still be required as the user story solidifies and experience is accumulated.

Nonetheless, it is still an interesting new development!

geo/gdml

This release adds preliminary support for parsing Geometry Description Markup Language (GDML) files, a de facto standard for describing (detector) geometries. The documentation for this new package is here: geo/gdml.

Help wanted and (gladly) accepted to get this package in a shape where it could be used for detailed detector studies! This is tracked here:

https://github.com/go-hep/hep/projects/4

hplot

hplot was slightly updated to cope with an interesting development percolating from upstream gonum/plot, namely: the migration to a new PDF backend that allows to embed fonts inside the output PDF file. No more PDFs that display weirdly on foreign computer. Yay!

This obviously means the resulting PDF files may be quite larger than with previous versions. (You can't have your cake and eat it.) You can use vgpdf.Canvas.EmbedFonts to get the old behaviour.

rootio

This release adds preliminary support for chaining multiple rootio.Trees into a logical view: the famed rootio.Chain.

Mohamed Amine El Gnaoui (a.k.a @maloft), our new summer student @LPC-Clermont, provided the initial implementation: thanks! More tests and benchmarks improvements yet to come :)

Another noteworthy change: rootio/cmd/root-srv dropped its dependency against github.com/satori/go.uuid in favor of github.com/pborman/uuid. The latter exposes a more stable API.

xrootd

This release adds yet another new package: xrootd. This package will provide (eventually) a pure-Go implementation of an XRootD client as well as a server.

Mikhail Ivchenko (a.k.a @EgorMatirov), our Google Summer of Code 2018 student has been already hard at work, providing support for:

the initial xrootd client,
the handshake with an XRootD-compliant server (C++ or otherwise), and
the protocol and login requests/responses.

The dirlist request/response is already in the pipe.

Scientific Software - Peer-reviewed - Go
Published by sbinet about 8 years ago

Go-HEP - Release v0.11

Release v0.11 is fresh from the oven.

This release drops official support for Go-1.7 and earlier, adds support for Go-1.10.

This allowed to migrate the fwk package away from golang.org/x/net/context. fwk now directly uses the context.Context type from the standard library.

rootio

This release brings quite a few improvements in the rootio area:

add support for empty ROOT files,
add support for reading remote ROOT files over HTTP. This is implemented in a quite naive way, using net/http.Get to download the whole file under a temporary directory. See go-hep/hep#142 for ideas on how to improve this,
add support for TH1-v6,
add support for streamer-less TDirectoryFile,
add support for TBranchElements in cmd/root-dump,
add support for displaying TH1, TH2 and TGraph{,Error}s in cmd/root-dump,
add support for branches and leaves with names only differing by case in cmd/root-gen-datareader. Now, the struct type being generated contains fields whose names start with ROOT_. The original branch or leaf name is then appended after ROOT_, unmodified.
add support for TKeys with a large binary payload,
add preliminary support for creating new Go types at runtime directly from a StreamerInfo value,
add more documentation about ROOT binary file format (TFile, TKey and a bit about TStreamerInfo so far.)

hbook

fixed a bug in the YODA ASCII file parsing code that would choke on analysis objects (histograms, scatters, ...) that contain whitespace.

AOB

LPC-Clermont has funded a 5 months internship student, starting now. Mohamed Amine El Gnaoui (@maloft) will:

implement reading all ROOT files created with ROOT-6,
implement writing ROOT files, in a ROOT-6 compatible way,
improve the read/write performances of Go-HEP to be on par with that of ROOT/C++ (using the builtin performance tools of the Go toolchain: pprof, execution tracer, ... and/or linux perf)
extend the test suite of Go-HEP for reading and writing ROOT files,
add benchmark tests for reading and writing ROOT files,
document the ROOT file format as understood by Go-HEP.

Welcome to Go-HEP, Amine!

Scientific Software - Peer-reviewed - Go
Published by sbinet about 8 years ago

Go-HEP - Release v0.10

Release v0.10 is fresh from the oven.

This release brings quite a few improvements in the rootio area:

support for ROOT files compressed with LZMA
support for ROOT files compressed with LZ4
support scanning ROOT trees with branches holding multiple leaves
support for ROOT trees holding bool, [X]bool and std::vector<bool>
support for ROOT files holding TBranch version 10
support for ROOT files holding TBranchElement version 8
support for ROOT files holding TH1 version 5
support for ROOT trees and branches produced with ROOT-6.12

The rootio/cmd/root-srv command dropped support for Go-1.6 and earlier. Please upgrade to a newer Go version.

In other news, the csvutil/csvdriver and hbook/ntup/ntcsv gained the ability to read remote CSV files (over HTTP.)

AOB

Better support for TStreamerElements and thus the ability to read nested std::vector<std::vector<T>> data or the ability to run root-dump and root-diff on all ROOT files, has started.

Google Summer of Code has started and Go-HEP is part of it, thanks to the CERN-HSF umbrella. Check out the 2 proposals:

Stay tuned.

Scientific Software - Peer-reviewed - Go
Published by sbinet over 8 years ago

Go-HEP - Release v0.9

Release v0.9 is fresh from the oven.

This is a quick and simple release fix with the following main features:

fix a broken link to the PDF LaTeX example (back from the multi- to mono-repo migration)
fix a compilation failure in rootio/cmd/root-srv originating from an API modification in a 3rd-party dependency (github.com/satori/go.uuid)

One interesting new feature is the introduction of a simple ROOT::TTree to CSV file conversion command: root2csv.

``` $> go get go-hep.org/x/hep/cmd/root2csv $> root2csv --help root2csv -h Usage of root2csv: -f string path to input ROOT file name -o string path to output CSV file name (default "output.csv") -t string name of the tree to convert (default "tree")

$> go doc go-hep.org/x/hep/cmd/root2csv root2csv converts the content of a ROOT TTree to a CSV file.

Usage of root2csv:
  -f string
            path to input ROOT file name
  -o string
            path to output CSV file name (default "output.csv")
  -t string
            name of the tree to convert (default "tree")

By default, root2csv will write out a CSV file with ';' as a column delimiter. root2csv ignores the branches of the TTree that are not supported by CSV:

- slices/arrays
- C++ objects

Example:

$> root2csv -o out.csv -t tree -f testdata/small-flat-tree.root
$> head out.csv
## Automatically generated from "testdata/small-flat-tree.root"
Int32;Int64;UInt32;UInt64;Float32;Float64;Str;N
0;0;0;0;0;0;evt-000;0
1;1;1;1;1;1;evt-001;1
2;2;2;2;2;2;evt-002;2
3;3;3;3;3;3;evt-003;3
4;4;4;4;4;4;evt-004;4
5;5;5;5;5;5;evt-005;5
6;6;6;6;6;6;evt-006;6
7;7;7;7;7;7;evt-007;7

```

AOB

The next release cycle will see improvements in the rootio department (read and write.)

Stay tuned.

Scientific Software - Peer-reviewed - Go
Published by sbinet over 8 years ago

Go-HEP - Release v0.8

Release v0.8 is fresh from the oven.

No API changes in go-hep itself, just a few modifications to cater with changes in gonum/v1/gonum and gonum/v1/plot:

gonum/v1/gonum/stat/distuv distributions replaced the name of their rand.Rand source field from Source to Src (to align with the standard library)
gonum/v1/gonum/stat/... packages now use golang.org/x/exp/rand.Rand instead of math/rand.Rand

The biggest news (and where the work mostly happened between v0.7 and v0.8) is that there's now a Jupyter instance with a Go kernel installed with Go-HEP and Gonum libraries pre-installed. This has been packaged up at go-hep/binder. You can try it there:

fmom

fmom gained a new top-level function fmom.InvMass(p1, p2 P4) float64 : go // InvMass computes the invariant mass of two incoming 4-vectors p1 and p2. func InvMass(p1, p2 P4) float64 { p := Add(p1, p2) return p.M() }

hplot

hplot gained a new top-level function Show:

go // Show displays the plot according to format, returning the raw bytes and // an error, if any. // // If format is the empty string, then "png" is selected. // The list of accepted format strings is the same one than from // the gonum.org/v1/plot/vg/draw.NewFormattedCanvas function. func Show(p *Plot, w, h vg.Length, format string) ([]byte, error) { ... }

Show is especially useful for the new support for Go in Jupyter notebooks (via the Neugram interpreter)

See https://github.com/go-hep/binder for more details and examples on using Go and Go-HEP in Jupyter.

David Blyth provided a new hplot.HInfoStdDev HInfoStyle value and made them bitset-like. Thanks David!

rootio

cmd/root-dump can now recursively handle rootio.Directory and thus walk an entire ROOT file.
cmd/root-ls got the same handling of rootio.Directory.

Scientific Software - Peer-reviewed - Go
Published by sbinet over 8 years ago

Go-HEP - Release v0.7

Release v0.7 is fresh from the oven with a few breaking API changes.

cmd/yoda

The cmd/rio2yoda, cmd/root2yoda and cmd/yoda2rio commands now support gzipped YODA files.

The cmd/yoda2rio is also more lenient when dealing with YODA objects that are not yet supported on the hbook side (Counter, S1D, ...) (thanks Lukas Heinrich for the report.)

fastjet

Bastian added the ability to plot the Voronoi diagram from a set of 2D-points.

hbook

hbook/yodacnv was modified to support the more lenient approach with regard to unsupported (yet!) YODA objects.

hplot

hplot has seen the most user-facing work:

a new default style that is more in-line with current aesthetic standards (ie: matplotlib-like)
hplot.New uses this new default style and thus is now able to ensure it won't fail loading the fonts
hplot.New thus only returns *hplot.Plot, without an error value
hplot.NewH1D is also able to ensure no fonts-loading error will araise and thus only return a *hplot.H1D

So, where you were doing:

```go p, err := hplot.New() if err != nil { log.Fatal(err) }

h, err := hplot.NewH1D(h1d) if err != nil { log.Fatal(err) } ```

you can now just write:

go p := hplot.New() h := hplot.NewH1D(h1d)

pawgo

pawgo has been slightly updated to support plotting 2-dim histograms. pawgo was also fixed to correctly handle YODA files converted to RIO (thanks Lukas Heinrich for the report.)

rootio

rootio gained 2 new commands:

rootio/cmd/root-diff: a command to print the differences between 2 ROOT files, including the content of their TTrees,
rootio/cmd/root-print: a command to print histograms contained in ROOT files into PDF, PNG, ... files.

rootio now also defines and exports the rootio.H1 and rootio.H2 interfaces that are implemented by the TH1x and TH2x (respectively) concrete types.

Scientific Software - Peer-reviewed - Go
Published by sbinet over 8 years ago

Go-HEP - Release v0.6

Release v0.6 is a small iteration (code-wise) but contains:

a software paper submitted to the Journal of OpenSource Software (JOSS):
a new WIP command, rootio/cmd/root-dump, to dump the content of ROOT files, including the entries of TTrees,
documentation fixes and updates in the rootio package,
still more work in the Delaunay and Voronoi area (thanks Bastian!)

One API change that is worth noting, is the retrieval of rootio.Keys from a rootio.Directory. The d4823f0 changes:

go type Directory interface { Get(namecycle string) (Object, bool) }

to:

go type Directory interface { Get(namecycle string) (Object, error) }

this allows to more quickly bubble-up errors while reducing boilerplate code at the Directory.Get(...) call site:

```diff diff --git a/cmd/root2npy/main.go b/cmd/root2npy/main.go index c81e81d..e3827fa 100644 --- a/cmd/root2npy/main.go +++ b/cmd/root2npy/main.go @@ -159,9 +159,9 @@ func main() { } defer f.Close()

obj, ok := f.Get(*tname)
if !ok {
log.Fatalf("no object named %q in file %q", *tname, *fname)
obj, err := f.Get(*tname)
if err != nil {
log.Fatal(err) } ```

Scientific Software - Peer-reviewed - Go
Published by sbinet over 8 years ago

Go-HEP - Release v0.5

Release v0.5 has seen some interesting work.

General

Go-HEP has migrated all its Gonum imports to use "gonum.org/v1/gonum/{mat,stat,...}", instead of the old "github.com/gonum/{mat64,stat,...}"
Go-HEP has also migrated all its use of gonum/plot from "github.com/gonum/plot/..." to the new "gonum.org/v1/plot/..."
Go-1.6's support has been dropped from the Travis-CI matrix
Go-1.9's support has been added to the Travis-CI matrix

rootio

The rootio package has seen most of the work: performance improvements for reading ROOT files, a new root2npy command and the beginning of the work to provide write support.

The performances of rootio have been assessed for the ACAT conference. The talk has all the details and can be read here. In a nutshell, rootio is now "just" 2 times slower than C++ ROOT to read files, and uses less than 10 times the VMem of C++ ROOT. More work to match the read speed of C++ ROOT is of course needed but we think we know how to recoup most of performance delta.

cmd/root2npy is a simple command to quickly convert a ROOT file containing a TTree into a NumPy array data file format.

cmd/root-srv has been updated to support https (thanks Michaël Ughetto).

cmd/root2yoda is a simple command to quickly convert a ROOT file containing histograms and profiles into a YODA file.

hbook & csvutil

csvutil/csvdriver has been improved to correctly handle CSV headers. You need to tell the driver that the CSV file contains a header describing the columns of the CSV file, but once that's done, you can use the correct columns' names in your SQL queries.

hbook/ntup/ntcsv is a new package that leverages the new feature of csvdriver to easily create hbook/ntup.Ntuple values from a CSV file, with the correct columns' names.

fit

Initial support for automatically computing the Hessian matrix has been added. If the user does not provide a Hess function, Gonum's diff/fd.Hessian will be used

fastjet

More of the GSoC work for the faster jet clustering has been integrated. Thanks Bastian Wieck.

AOB

With release v0.5, Go-HEP gained its first external user: eicio, "an exploratory project testing ideas about forward-looking IO solutions to suit the needs of the Electron Ion Collider (EIC) community." Welcome eicio!

Scientific Software - Peer-reviewed - Go
Published by sbinet over 8 years ago

Go-HEP - Release v0.4

Release v0.4 is a small iteration (because of holidays.)

But still, it does bring interesting things to the table:

root2npy, a new command to convert the content of simple TTrees into a NumPy data file,
migration from the github.com/gonum/xyz packages to their new location -- gonum.org/v1/gonum/xyz,

`fastjet`

preliminary work for implementing better/faster jet clustering, thanks Bastian Wieck

`hplot`

make the hplot.H1D type implement gonum/plot.Thumbnailer (for better integration with legends and labels)

`lcio`

fix a bug in the marshaling of lcio.TrackerRawData, thanks David Blyth

Scientific Software - Peer-reviewed - Go
Published by sbinet almost 9 years ago

Go-HEP - Release v0.3

Release v0.3 brings read support for ROOT TTrees containing user-defined classes.

Not everything is supported, though.

`rootio`

What is supported and tested so far (in no-split and full-split mode):

https://github.com/go-hep/hep/blob/v0.3/rootio/gendata/gen-evnt-tree.go#L51 c++ struct P3 { int32_t Px; double Py; int32_t Pz; }; struct Event { TString Beg; int16_t I16; int32_t I32; int64_t I64; uint16_t U16; uint32_t U32; uint64_t U64; float F32; double F64; TString Str; P3 P3; int16_t ArrayI16[ARRAYSZ]; int32_t ArrayI32[ARRAYSZ]; int64_t ArrayI64[ARRAYSZ]; uint16_t ArrayU16[ARRAYSZ]; uint32_t ArrayU32[ARRAYSZ]; uint64_t ArrayU64[ARRAYSZ]; float ArrayF32[ARRAYSZ]; double ArrayF64[ARRAYSZ]; int32_t N; int16_t *SliceI16; //[N] int32_t *SliceI32; //[N] int64_t *SliceI64; //[N] uint16_t *SliceU16; //[N] uint32_t *SliceU32; //[N] uint64_t *SliceU64; //[N] float *SliceF32; //[N] double *SliceF64; //[N] std::string StdStr; std::vector<int16_t> StlVecI16; std::vector<int32_t> StlVecI32; std::vector<int64_t> StlVecI64; std::vector<uint16_t> StlVecU16; std::vector<uint32_t> StlVecU32; std::vector<uint64_t> StlVecU64; std::vector<float> StlVecF32; std::vector<double> StlVecF64; std::vector<std::string> StlVecStr; TString End; };

which can be read with a Go struct with the same layout, translating std::vector<T> into slices of T, std::string and TString into Go string, etc... see: https://github.com/go-hep/hep/blob/v0.3/rootio/tree_test.go#L59 ``go type EventType struct { Evt EventDatarootio:"evt"` }

type Vec3 struct { X int32 rootio:"Px" Y float64 rootio:"Py" Z int32 rootio:"Pz" }

type EventData struct { Beg string rootio:"Beg" I16 int16 rootio:"Int16" I32 int32 rootio:"Int32" I64 int64 rootio:"Int64" U16 uint16 rootio:"UInt16" U32 uint32 rootio:"UInt32" U64 uint64 rootio:"UInt64" F32 float32 rootio:"Float32" F64 float64 rootio:"Float64" Str string rootio:"Str" Vec Vec3 rootio:"P3" ArrI16 [10]int16 rootio:"ArrayI16" ArrI32 [10]int32 rootio:"ArrayI32" ArrI64 [10]int64 rootio:"ArrayI64" ArrU16 [10]uint16 rootio:"ArrayU16" ArrU32 [10]uint32 rootio:"ArrayU32" ArrU64 [10]uint64 rootio:"ArrayU64" ArrF32 [10]float32 rootio:"ArrayF32" ArrF64 [10]float64 rootio:"ArrayF64" N int32 rootio:"N" SliI16 []int16 rootio:"SliceI16" SliI32 []int32 rootio:"SliceI32" SliI64 []int64 rootio:"SliceI64" SliU16 []uint16 rootio:"SliceU16" SliU32 []uint32 rootio:"SliceU32" SliU64 []uint64 rootio:"SliceU64" SliF32 []float32 rootio:"SliceF32" SliF64 []float64 rootio:"SliceF64" StdStr string rootio:"StdStr" VecI16 []int16 rootio:"StlVecI16" VecI32 []int32 rootio:"StlVecI32" VecI64 []int64 rootio:"StlVecI64" VecU16 []uint16 rootio:"StlVecU16" VecU32 []uint32 rootio:"StlVecU32" VecU64 []uint64 rootio:"StlVecU64" VecF32 []float32 rootio:"StlVecF32" VecF64 []float64 rootio:"StlVecF64" VecStr []string rootio:"StlVecStr" End string rootio:"End" } ```

support for more use cases will come in due time. Stay tuned!

`hbook`

Michael Ughetto contributed code to retrieve the bins of hbook.H2D histograms, thanks @mughetto

See: https://go-hep.org/news/release-0.3

Scientific Software - Peer-reviewed - Go
Published by sbinet about 9 years ago

Go-HEP - Release v0.2

new fastjet algorithms: EeGenKtAlgorithm, EeKtAlgorithm (thanks @Bastiantheone)
fastjet can now compute Exclusive jets (thanks @Bastiantheone)
new lcio package, with initial read/write support for a few types of the LCIO event data model, namely:
- MCParticle
- SimTrackerHit
- SimCalorimeterHit
- LCFloatVec
- LCIntVec
- LCStrVec
- RawCalorimeterHit
- CalorimeterHit
- TrackerHit
- TrackerHitPlane
- Track
- Cluster
- Vertex
- ReconstructedParticle
- LCGenericObject

(this limited set already allows to test the official LCIO C++ examples)

Next cycle will (really) see some work on hep/rootio.

See: https://go-hep.org/news/release-0.2

Scientific Software - Peer-reviewed - Go
Published by sbinet about 9 years ago

Go-HEP - Release v0.1

First official release of go-hep.

See the announcement:

https://go-hep.org/news/release-0.1

Scientific Software - Peer-reviewed - Go
Published by sbinet about 9 years ago

Recent Releases of Go-HEP

Go-HEP - Release v0.37.0

What's Changed

Go-HEP - Release v0.36.0

What's Changed

Go-HEP - Release v0.35.0

What's Changed

Go-HEP - Release v0.34.1

What's Changed

Go-HEP - Release v0.34.0

hplot

groot

Changelog

Go-HEP - Release v0.33.0

cmd

hplot

sliceop

Changelog

Go-HEP - Release v0.32.1

What's Changed

Go-HEP - Release v0.32.0

brio

fwk

groot

hbook

lhef

sliceop

Changelog

What's Changed

New Contributors

Go-HEP - Release v0.31.1

Go-HEP - Release v0.31.0

groot

xrootd

AOB

Changelog

Go-HEP - Release v0.30.1

fwk

groot

hplot

sliceop

AOB

Changelog

Go-HEP - Release v0.30.0

fwk

groot

hplot

sliceop

AOB

Changelog

Go-HEP - Release v0.29.1

arrow

groot

hbook

hplot

xrootd

AOB

Changelog

Go-HEP - Release v0.28.3

Go-HEP - Release v0.28.2

Go-HEP - Release v0.28.1

Go-HEP - Release v0.28.0: write structured-tree

csvutil

groot

hplot

AOB

ChangeLog

Go-HEP - Release v0.27.0: groot faster than ROOT

cmd

fit

fmom

groot

hbook

hplot

lhef

sliceop/f64s

AOB

Go-HEP - Release v0.26.0: rtree.Formula & hplot improvements

Go-HEP - Release v0.25.0: fmom.Boost, LHEF and groot.Reader

Go-HEP - Release v0.24.1: ROOT-6.20