hivemind - 1.11.11: Compatibility improvements, version upgrades

What's Changed

• Fix edge cases in (de)serializetorchtensor by @justheuristic in https://github.com/learning-at-home/hivemind/pull/591
• Fix deprecations and update dependencies for examples/albert by @mryab in https://github.com/learning-at-home/hivemind/pull/595
• Fix OptimizerWrapper creation, test gradient clipping by @mryab in https://github.com/learning-at-home/hivemind/pull/593
• Update petals homepage URL by @dpirad007 in https://github.com/learning-at-home/hivemind/pull/599
• Bump p2pd version by @dvmazur in https://github.com/learning-at-home/hivemind/pull/598
• Update Dockerfile, fix compatibility with PyTorch>=2.3.0 by @mryab in https://github.com/learning-at-home/hivemind/pull/614
• Override the logging level in testcliscripts by @mryab in https://github.com/learning-at-home/hivemind/pull/617
• Improve the process cleanup logic when running tests by @mryab in https://github.com/learning-at-home/hivemind/pull/616
• Add downloading time to averager by @samsja in https://github.com/learning-at-home/hivemind/pull/618
• Migrate to pydantic >= 2.0 by @Vectorrent in https://github.com/learning-at-home/hivemind/pull/613
• Update README.md dead link to HivemindStrategy by @cleong110 in https://github.com/learning-at-home/hivemind/pull/607
• Deprecate Python 3.7, switch to Python 3.11 in CI by @mryab in https://github.com/learning-at-home/hivemind/pull/621
• Limit Protobuf version by 5.28.0 by @mryab in https://github.com/learning-at-home/hivemind/pull/631
• Clean up resources in DHT/P2P, improve test robustness by @mryab in https://github.com/learning-at-home/hivemind/pull/636
• Clarify the download location of p2pd in setup.py by @xloem in https://github.com/learning-at-home/hivemind/pull/633
• Add CITATION.cff, update citations by @mryab in https://github.com/learning-at-home/hivemind/pull/630
• Remove unused code and imports by @mryab in https://github.com/learning-at-home/hivemind/pull/632
• Run CI on Modal, upgrade Bitsandbytes by @mryab in https://github.com/learning-at-home/hivemind/pull/641
• Add multiaddr as subpackage by @dvmazur in https://github.com/learning-at-home/hivemind/pull/637
• Fix a typo in the quickstart by @emiapwil in https://github.com/learning-at-home/hivemind/pull/627
• Enable GitHub Actions for tests by @mryab in https://github.com/learning-at-home/hivemind/pull/644
• Delete conditions for run-tests.yml by @mryab in https://github.com/learning-at-home/hivemind/pull/645
• Fix GradScaler import on torch >= 2.3.0 for benchmark_optimizer.py by @Vectorrent in https://github.com/learning-at-home/hivemind/pull/620
• Fix RemoteMixtureOfExperts and RemoteSwitchMixtureOfExperts backward() on GPU by @Vectorrent in https://github.com/learning-at-home/hivemind/pull/626
• Upgrade pytest to 8.3.5 by @mryab in https://github.com/learning-at-home/hivemind/pull/646
• Replace black+isort with Ruff by @mryab in https://github.com/learning-at-home/hivemind/pull/622
• Fix version-file for the Ruff action by @mryab in https://github.com/learning-at-home/hivemind/pull/649
• Make testaveragingcancel more robust by @mryab in https://github.com/learning-at-home/hivemind/pull/650

New Contributors

• @dpirad007 made their first contribution in https://github.com/learning-at-home/hivemind/pull/599
• @samsja made their first contribution in https://github.com/learning-at-home/hivemind/pull/618
• @Vectorrent made their first contribution in https://github.com/learning-at-home/hivemind/pull/613
• @cleong110 made their first contribution in https://github.com/learning-at-home/hivemind/pull/607
• @xloem made their first contribution in https://github.com/learning-at-home/hivemind/pull/633
• @emiapwil made their first contribution in https://github.com/learning-at-home/hivemind/pull/627

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.1.10...1.11.11

- Python
Published by mryab about 1 year ago

hivemind - 1.1.10: macOS and Linux ARM support

What's Changed

• [minor] allow overriding args/kwargs behavior in Runtime by @justheuristic in https://github.com/learning-at-home/hivemind/pull/587
• Use proper p2pd binary on macOS by @borzunov in https://github.com/learning-at-home/hivemind/pull/586
• Force DHT to be mp.context.ForkProcess by @borzunov in https://github.com/learning-at-home/hivemind/pull/589
• Use separate binaries for all supported OS and architectures by @borzunov in https://github.com/learning-at-home/hivemind/pull/588
• Consider multiple CPU arch aliases by @borzunov in https://github.com/learning-at-home/hivemind/pull/590

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.1.9...1.1.10

- Python
Published by mryab over 2 years ago

hivemind - 1.1.9: Patch release

What's Changed

• Improve handling of KeyboardInterrupt in CLI applications by @mryab in https://github.com/learning-at-home/hivemind/pull/567
• Measure coverage of subprocesses, exclude protobuf compiled files by @mryab in https://github.com/learning-at-home/hivemind/pull/568
• Require pydantic<2.0 unless it's supported by @borzunov in https://github.com/learning-at-home/hivemind/pull/573
• Support Python 3.11 by @borzunov @justheuristic in https://github.com/learning-at-home/hivemind/pull/574
• Fix using .lstrip() in hivemind.compression by @borzunov in https://github.com/learning-at-home/hivemind/pull/578
• Fix TypeError in P2P._terminate() by @borzunov in https://github.com/learning-at-home/hivemind/pull/579

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.1.8...1.1.9

- Python
Published by mryab almost 3 years ago

hivemind - 1.1.8: Patch release

What's Changed

• Fix errors in hivemind.p2p and hivemind.compression by @borzunov in https://github.com/learning-at-home/hivemind/pull/565

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.1.7...1.1.8

- Python
Published by mryab about 3 years ago

hivemind - 1.1.7: Patch release

What's Changed

• Remove direct coroutine call by @srogatch in https://github.com/learning-at-home/hivemind/pull/557
• Require torch<2.0 until 2.0 is supported, add Python 3.10 to CI by @borzunov in https://github.com/learning-at-home/hivemind/pull/558
• Support PyTorch 2.0.0 by @justheuristic in https://github.com/learning-at-home/hivemind/pull/559
• Fix bfloat16 serialization for tensors with zero elements by @borzunov in https://github.com/learning-at-home/hivemind/pull/560
• Allow RemoteExpertWorker run coroutines concurrently by @borzunov in https://github.com/learning-at-home/hivemind/pull/561
• Fix broken link, min torch version in readme by @borzunov in https://github.com/learning-at-home/hivemind/pull/562

New Contributors

• @srogatch made their first contribution in https://github.com/learning-at-home/hivemind/pull/557

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.1.6...1.1.7

- Python
Published by mryab about 3 years ago

hivemind - 1.1.6: Patch release

What's Changed

• Add docstrings for use_relay/use_auto_relay, add them to hivemind-cli by @borzunov in https://github.com/learning-at-home/hivemind/pull/547
• Fix docstrings about relays by @borzunov in https://github.com/learning-at-home/hivemind/pull/548
• New bitsandbytes (with latest GPU support) by @justheuristic in https://github.com/learning-at-home/hivemind/pull/554
• Improve bfloat16 serialization (backward compatible) by @justheuristic in https://github.com/learning-at-home/hivemind/pull/553
• Fix exception in MPFuture.del() by @justheuristic in https://github.com/learning-at-home/hivemind/pull/555

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.1.5...1.1.6

- Python
Published by mryab over 3 years ago

hivemind - 1.1.5: Patch release

What's Changed

• Make DecentralizedAverager resistant to KeyboardInterrupt by @ikmckenz in https://github.com/learning-at-home/hivemind/pull/530
• Switch to p2pd v0.3.14, libp2p v0.24.0 by @justheuristic in https://github.com/learning-at-home/hivemind/pull/531
• Fix typo in beam_search.py by @eltociear in https://github.com/learning-at-home/hivemind/pull/532
• Fix link to the papers in readme by @borzunov in https://github.com/learning-at-home/hivemind/pull/535
• Fix a broken URL in contributing.md by @Vahe1994 in https://github.com/learning-at-home/hivemind/pull/538
• Support circuit relay v2, add tests by @Vahe1994 in https://github.com/learning-at-home/hivemind/pull/537
• Fix new parameters in hivemind.P2PDaemon by @borzunov in https://github.com/learning-at-home/hivemind/pull/540
• Fix logging in Jupyter and Colab by @borzunov in https://github.com/learning-at-home/hivemind/pull/542
• Add comment warning for nonblocking + sharememory by @justheuristic in https://github.com/learning-at-home/hivemind/pull/541
• Import bitsandbytes only if it is used by @borzunov in https://github.com/learning-at-home/hivemind/pull/546
• Add Codespell to CI, fix typos by @mryab in https://github.com/learning-at-home/hivemind/pull/543

New Contributors

• @ikmckenz made their first contribution in https://github.com/learning-at-home/hivemind/pull/530
• @eltociear made their first contribution in https://github.com/learning-at-home/hivemind/pull/532
• @Vahe1994 made their first contribution in https://github.com/learning-at-home/hivemind/pull/538

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.1.4...1.1.5

- Python
Published by mryab over 3 years ago

hivemind - 1.1.4: Patch release

What's Changed

• Update p2pd to v0.3.13 by @borzunov in https://github.com/learning-at-home/hivemind/pull/527

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.1.3...1.1.4

- Python
Published by mryab over 3 years ago

hivemind - 1.1.3: Patch release

What's Changed

• Update moe.md by @cirquit in https://github.com/learning-at-home/hivemind/pull/516
• Fix "unable to open shared memory" while using MPFuture by @borzunov in https://github.com/learning-at-home/hivemind/pull/517
• Fix MPFuture failing outside inference mode by @borzunov in https://github.com/learning-at-home/hivemind/pull/521
• Bump torch to >=1.9.0 by @borzunov in https://github.com/learning-at-home/hivemind/pull/522
• Fix P2PDaemon's idle timeout by @borzunov in https://github.com/learning-at-home/hivemind/pull/523
• Support torch.bfloat16 in hivemind.compression by @borzunov in https://github.com/learning-at-home/hivemind/pull/524
• Remove stale PeerIDs in hivemind-dht's routing table by @borzunov in https://github.com/learning-at-home/hivemind/pull/525

New Contributors

• @cirquit made their first contribution in https://github.com/learning-at-home/hivemind/pull/516

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.1.2...1.1.3

- Python
Published by mryab over 3 years ago

hivemind - 1.1.2: Patch release

What's Changed

• Forbid protobuf 4.x in requirements by @justheuristic in https://github.com/learning-at-home/hivemind/pull/508
• Check if identity is already taken by @borzunov in https://github.com/learning-at-home/hivemind/pull/511
• Add Petals to "Example Use Cases" by @borzunov in https://github.com/learning-at-home/hivemind/pull/512
• Follow up #501 and #511 with minor fixes by @borzunov in https://github.com/learning-at-home/hivemind/pull/513
• Update bitsandbytes, relax its version constraint by @mryab in https://github.com/learning-at-home/hivemind/pull/510

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.1.1...1.1.2

- Python
Published by mryab over 3 years ago

hivemind - 1.1.1: Patch release, bitsandbytes support for compression

What's Changed

• Handle errors in Runtime by @justheuristic in https://github.com/learning-at-home/hivemind/pull/489
• metadata type changed to bytes by @GreenFatGuy in https://github.com/learning-at-home/hivemind/pull/491
• fix: Parameter Averaging quickstart clarification by @IAL32 in https://github.com/learning-at-home/hivemind/pull/492
• Make DHT ignore SIGINT by @dbaranchuk in https://github.com/learning-at-home/hivemind/pull/493
• Update README with latest projects and publications by @mryab in https://github.com/learning-at-home/hivemind/pull/494
• Add links to "Example Use Cases" by @borzunov in https://github.com/learning-at-home/hivemind/pull/497
• Support bfloat16 for autograd by @dbaranchuk in https://github.com/learning-at-home/hivemind/pull/499
• Remove libp2p handlers when ConnectionHandler, DHT, and DecentralizedAverager are shut down by @borzunov in https://github.com/learning-at-home/hivemind/pull/501
• Fix PyTorch warning suppression by @borzunov in https://github.com/learning-at-home/hivemind/pull/502
• Fix a potential deadlock in await_asynchronously with nested locks by @justheuristic in https://github.com/learning-at-home/hivemind/pull/503
• Require TaskPoolBase to implement loadbatchto_runtime by @justheuristic in https://github.com/learning-at-home/hivemind/pull/506
• Change runtime.py to choose tasks with lowest (instead of highest) priority by @justheuristic in https://github.com/learning-at-home/hivemind/pull/505
• Add support for quantization with bitsandbytes by @mryab in https://github.com/learning-at-home/hivemind/pull/490

New Contributors

• @IAL32 made their first contribution in https://github.com/learning-at-home/hivemind/pull/492
• @dbaranchuk made their first contribution in https://github.com/learning-at-home/hivemind/pull/493

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.1.0...1.1.1

- Python
Published by mryab over 3 years ago

hivemind - 1.1.0: libp2p support in hivemind.moe, PowerSGD training, persistent P2P identity

Release highlights

• Starting from this release, all components of hivemind.moe use libp2p for communication. This comes with the same benefits as in averaging and DHT previously (simplified NAT traversal, better performance, etc.) and marks the end of gRPC usage in hivemind. The user API is mostly the same: if you were using abstractions like RemoteMixtureOfExperts, the code should not be changed, although cross-release training is not possible.
• If you need another way to reduce the network footprint during training with hivemind.Optimizer, you can now use PowerSGD for gradient averaging. This method decreases the communication costs by factorizing the gradients of the model and aggregating the factorized versions. To enable this method in your code, pass grad_averager_factory=partial(PowerSGDGradientAverager, averager_rank=RANK) when creating an instance of Optimizer. Here, RANK denotes the factorization rank; lower values give higher compression at the cost of the reconstruction quality.
• Similarly to hivemind-server, it is now possible to launch a dedicated DHT instance with a command-line tool. The tool, available via hivemind-dht, can be used to quickly create a lightweight peer that is used mostly for connecting others to the DHT (for example, on a publicly available server) or for DHT metadata replication.
• Previously, restarting a libp2p instance required generating a new P2P identity, which resulted in a new multiaddress. Thus, it was difficult to use the same command to connect to a peer in case of repeated launches, which is often the case during debugging. Now, you can store the persistent peer identity of a peer in a file and reuse it between launches: this is done by specifying the --identity_path argument, available both in the ALBERT example and CLI tools of hivemind.

Deprecations

• The parameters quic, use_relay_hop, and use_relay_discovery of hivemind.P2P are deprecated since our update of the libp2p dependency in the p2p daemon. They will be removed in the 1.2.0 release of hivemind

What's Changed

• Pin pytest version in requirements-dev, use file_descriptor in tests by @justheuristic in https://github.com/learning-at-home/hivemind/pull/454
• Pin isort version, bump black by @mryab in https://github.com/learning-at-home/hivemind/pull/456
• Clean compression/init.py by @borzunov in https://github.com/learning-at-home/hivemind/pull/460
• Do not use offloadoptimizer with localupdates by deafult by @foksly in https://github.com/learning-at-home/hivemind/pull/462
• Add PowerSGD for compressed gradient averaging by @artek0chumak in https://github.com/learning-at-home/hivemind/pull/432
• Bump Black to 22.3.0, pin Golang version by @mryab in https://github.com/learning-at-home/hivemind/pull/466
• uselocalupdates in optimizer by @justheuristic in https://github.com/learning-at-home/hivemind/pull/468
• Update p2pd to v0.3.8 (and libp2p to v0.17.0) by @borzunov in https://github.com/learning-at-home/hivemind/pull/469
• Generate new private key if identity file doesn't exist by @borzunov in https://github.com/learning-at-home/hivemind/pull/473
• Convert hivemind.server to libp2p backend by @GreenFatGuy in https://github.com/learning-at-home/hivemind/pull/470
• Implement a CLI for hivemind.DHT by @mryab in https://github.com/learning-at-home/hivemind/pull/465
• Use PeerID exclusively to address MoE experts by @justheuristic in https://github.com/learning-at-home/hivemind/pull/479
• Remove deprecated code in hivemind.optim and hivemind.averaging before the 1.1.0 release by @mryab in https://github.com/learning-at-home/hivemind/pull/480
• Fix shape validation in GradientAverager by @mryab in https://github.com/learning-at-home/hivemind/pull/481
• Change expiration time in declareexperts, fix updateperiod discrepancy by @justheuristic in https://github.com/learning-at-home/hivemind/pull/482
• Add identity_path option for MoE.Server runners by @GreenFatGuy in https://github.com/learning-at-home/hivemind/pull/484
• Simplify ExpertBackend interface by @justheuristic in https://github.com/learning-at-home/hivemind/pull/483
• Clean up imports, remove unused utils by @mryab in https://github.com/learning-at-home/hivemind/pull/486
• finish renaming experts -> module_backends in ConnectionHandler by @justheuristic in https://github.com/learning-at-home/hivemind/pull/487
• Remove gRPC services and grpcio requirement by @mryab in https://github.com/learning-at-home/hivemind/pull/485

New Contributors

• @GreenFatGuy made their first contribution in https://github.com/learning-at-home/hivemind/pull/470

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.0.1...1.1.0

- Python
Published by mryab almost 4 years ago

hivemind - 1.0.1: Patch release

What's Changed

• Improve user-friendliness and fix misc errors in Optimizer, Averager and P2P by @justheuristic @pr-Mais @borzunov @mrseeker @mryab in https://github.com/learning-at-home/hivemind/pull/428
• Skip gradient averaging if there are no other peers by @justheuristic @soodoshll @borzunov in https://github.com/learning-at-home/hivemind/pull/440
• Move hivemind.Server from init, streamline imports by @mryab in https://github.com/learning-at-home/hivemind/pull/441
• Change makeempty to makezeros for TensorDescriptor by @mryab in https://github.com/learning-at-home/hivemind/pull/442
• Fix offloaded optimizer with single peer by @justheuristic @elricwan @borzunov in https://github.com/learning-at-home/hivemind/pull/450
• Fix "too many open files" issue by @yhn112 in https://github.com/learning-at-home/hivemind/pull/444

Full Changelog: https://github.com/learning-at-home/hivemind/compare/1.0.0...1.0.1

- Python
Published by mryab over 4 years ago

hivemind - 1.0.0: hivemind.Optimizer, improved averaging stability, better logging

What's Changed

• Fix averager speed for TCP connections by @borzunov in https://github.com/learning-at-home/hivemind/pull/373
• Fix "Too many open files" and load state freezing by @justheuristic in https://github.com/learning-at-home/hivemind/pull/371
• Prefetch while reading rpcaggregatepart() outputs by @borzunov in https://github.com/learning-at-home/hivemind/pull/370
• Use ModeClient in libp2p DHT in case of --client_mode by @borzunov in https://github.com/learning-at-home/hivemind/pull/374
• Integrate p2pd logs and outputs into hivemind logging by @borzunov in https://github.com/learning-at-home/hivemind/pull/375
• Split compression strategies into separate classes by @justheuristic in https://github.com/learning-at-home/hivemind/pull/366
• Implement colored logs by @borzunov in https://github.com/learning-at-home/hivemind/pull/377
• Parametrize max message size for persistent connections by @deniskamazur in https://github.com/learning-at-home/hivemind/pull/376
• Make log handlers configurable, shorten entries by @borzunov in https://github.com/learning-at-home/hivemind/pull/378
• Enable log handler in benchmarks and run_server by @borzunov in https://github.com/learning-at-home/hivemind/pull/380
• Fix step_tolerance in CollaborativeOptimizer by @justheuristic in https://github.com/learning-at-home/hivemind/pull/383
• Fix pickle vulnerability by @deniskamazur in https://github.com/learning-at-home/hivemind/pull/386
• Remove arguments with default values from example instructions by @borzunov in https://github.com/learning-at-home/hivemind/pull/388
• Implement weight as part of the allreduce protocol, not matchmaking by @justheuristic in https://github.com/learning-at-home/hivemind/pull/384
• Support different AMP & buffer configurations in one experiment, fix minor bugs by @justheuristic in https://github.com/learning-at-home/hivemind/pull/389
• Fix codecovindevelop_mode with pip>=21.2 by @justheuristic in https://github.com/learning-at-home/hivemind/pull/393
• Fix minor issues in documentation by @borzunov in https://github.com/learning-at-home/hivemind/pull/392
• Apply averager updates asynchronously by @justheuristic in https://github.com/learning-at-home/hivemind/pull/395
• Fix schema typing by @justheuristic in https://github.com/learning-at-home/hivemind/pull/396
• backport PerformanceEMA from serversideaveraging by @justheuristic in https://github.com/learning-at-home/hivemind/pull/397
• Add an option to pre-schedule averaging by @justheuristic in https://github.com/learning-at-home/hivemind/pull/398
• Move DHT to dht/dht.py, update DHT figure by @justheuristic in https://github.com/learning-at-home/hivemind/pull/399
• [hotfix] replace StepControl.canmodify with beganallreduce by @justheuristic in https://github.com/learning-at-home/hivemind/pull/402
• move PerformanceEMA to utils, TrainingAverager to optim, update utils by @justheuristic in https://github.com/learning-at-home/hivemind/pull/405
• Add GradientAverager with support for delayed averaging by @justheuristic in https://github.com/learning-at-home/hivemind/pull/404
• [hivemind.Optimizer] TrainingStateAverager by @justheuristic in https://github.com/learning-at-home/hivemind/pull/407
• Catch OSError in MPFuture by @artek0chumak in https://github.com/learning-at-home/hivemind/pull/409
• [hivemind.Optimizer] ProgressTracker by @justheuristic in https://github.com/learning-at-home/hivemind/pull/408
• Fix minor bugs in GradientAverager by @justheuristic in https://github.com/learning-at-home/hivemind/pull/410
• Make target group size optional by @justheuristic in https://github.com/learning-at-home/hivemind/pull/412
• Prepare GradScaler for hivemind.Optimizer by @justheuristic in https://github.com/learning-at-home/hivemind/pull/413
• Patch recursive cancel in StepControl by @justheuristic in https://github.com/learning-at-home/hivemind/pull/411
• Replace the invalid link to discord by @artek0chumak in https://github.com/learning-at-home/hivemind/pull/414
• Implement state sharing priority by @justheuristic in https://github.com/learning-at-home/hivemind/pull/415
• Implement core functionality of hivemind.Optimizer by @justheuristic in https://github.com/learning-at-home/hivemind/pull/403
• DHT Benchmark with asynchronous w/r by @MuXauJl11110 in https://github.com/learning-at-home/hivemind/pull/406
• Hotfix: loadstatefrompeers with offloadoptimizer by @justheuristic in https://github.com/learning-at-home/hivemind/pull/417
• Improve Optimizer docs, update quickstart to use Optimizer by @justheuristic in https://github.com/learning-at-home/hivemind/pull/416
• Quickstart: typos and references by @justheuristic in https://github.com/learning-at-home/hivemind/pull/420
• Remove trailing dots in log messages and errors by @borzunov in https://github.com/learning-at-home/hivemind/pull/419
• Do not log caller for INFO messages by @borzunov in https://github.com/learning-at-home/hivemind/pull/418
• Improve hivemind.optim.experimental and averager stability by @borzunov in https://github.com/learning-at-home/hivemind/pull/421
• Add minor tweaks learned from the NeurIPS demo run by @justheuristic in https://github.com/learning-at-home/hivemind/pull/422
• Improve All-Reduce fault-tolerance by @justheuristic in https://github.com/learning-at-home/hivemind/pull/423
• Fix All-Reduce fault-tolerance: catch Exception instead of BaseException by @justheuristic in https://github.com/learning-at-home/hivemind/pull/424
• Fix Task was destroeyd but is pending (put items) by @justheuristic in https://github.com/learning-at-home/hivemind/pull/427
• Use hivemind.Optimizer in examples/albert by @mryab in https://github.com/learning-at-home/hivemind/pull/426

New Contributors

• @artek0chumak made their first contribution in https://github.com/learning-at-home/hivemind/pull/409
• @MuXauJl11110 made their first contribution in https://github.com/learning-at-home/hivemind/pull/406

Full Changelog: https://github.com/learning-at-home/hivemind/compare/0.10.0...1.0.0

- Python
Published by mryab over 4 years ago

hivemind - 0.10.0: libp2p-based averaging, improved P2P daemon performance

This release contains the following new features and bugfixes:

• Fix deadlocks in DecentralizedAverager and MPFuture (#331) (@borzunov @justheuristic)
• Resolve deadlock in MPFuture (#337) (@justheuristic @borzunov @yhn112)
• Convert averager to libp2p backend (#323) (@borzunov @mryab)
• Refactor naming and serialization for PeerIDs (#339) (@borzunov)
• Set default DHT num_workers = 4 (#342) (@borzunov @deniskamazur @justheuristic @mryab)
• Fix typo in dht.md (#345) (@justheuristic)
• Fix some warnings related to asyncio (#346) (@borzunov)
• Speed up P2P client creation (#343) (@deniskamazur @borzunov)
• Propagate startup errors from DHT and averager processes (#347) (@borzunov)
• Add less comparator for PeerID (#353) (@deniskamazur @borzunov)
• Fix minor asyncio issues in averager (#356) (@borzunov @justheuristic)
• Optimize unary handlers with persistent connections to P2P daemon (#328) (@deniskamazur)
• Fix import error breaking AllReduceRunner.senderrortopeer() (#360) (@borzunov)
• Fix logger warning in P2P (#361) (@borzunov)
• Disable QUIC (#355) (@borzunov)
• Disable elasticity for averaging, add error handling (#362) (@justheuristic @mryab)
• Improve Matchmaking finalizers (#357) (@borzunov)
• Allow to specify P2P identity file (#363) (@borzunov)
• Fix loglevel for a message in readfrompersistentconn() (#364) (@borzunov)

- Python
Published by mryab almost 5 years ago

hivemind - 0.9.10: libp2p-based DHT, per-tensor compression, better documentation

This release contains the following features and bugfixes: * Add p2pd to packagedata (#287) (@mryab) * Add per-tensor compression, make All-Reduce faster and more flexible (#272) (@justheuristic @mponty @mryab @yhn112 @borzunov) * Fix race condition while reserving ports in P2P (#299) (@borzunov) * Add graceful shutdown to DHT and Averager (#301) (@justheuristic @mryab) * Make checkpointing optional in example (#303) (@yhn112) * Refactor MPFuture to use a single pipe/thread per process (#298) (@justheuristic @borzunov @mryab @yhn112) * Split hivemind.client into hivemind.averaging and hivemind.moe (#304) (@mryab) * Update readthedocs with hivemind.optim (#288) (@yhn112 @justheuristic) * Minor fixes in examples/albert (#308) (@yhn112) * Upload the model with pushto_hub in examples (#297) (@leshanbog @mryab @justheuristic) * Account for multi-gpu devices in examples/albert (#309) (@justheuristic) * Convert DHT to libp2p backend (#296) (@borzunov @skobellev) * Simplify argument parsing, update docs in ALBERT example (#315) (@mryab @justheuristic @yhn112) * Improve P2P handler throughput and interface (#316) (@borzunov) * Remove shared memory from MPFuture, fix minor bugs (#317) (@justheuristic @borzunov @mryab) * Implement protobuf-based stream handlers over libp2p backend (#318) (@borzunov) * Refactor for v0.9.10 and fix example (#319) (@justheuristic @borzunov) * Update quickstart tutorials and acknowledgements (#307) (@justheuristic @yhn112 @borzunov @mryab)

- Python
Published by mryab almost 5 years ago

hivemind - 0.9.9: Improved libp2p installation, auxiliary peers support, logging in benchmarks

This release contains the following improvements and bugfixes:

• Add relay options to P2P (#268) (@deniskamazur)
• Add packaging to requirements (#269) (@deniskamazur)
• Disable p2pd compilation by default (#270) (@yhn112 @justheuristic)
• Measure testing coverage on pull request (#271) (@yhn112)
• Update p2pd md5 checksum (#273) (@deniskamazur)
• Use logging in benchmarks, fix libp2p-related issues (#280) (@justheuristic)
• Add BibTeX reference for the library to README (#283) (@mryab)
• Fix Codecov (#282) (@yhn112)
• Remove use of packaging module (#284) (@borzunov)
• Support auxiliary peers in CollaborativeOptimizer (#279) (@yhn112 @justheuristic @mryab)

- Python
Published by mryab almost 5 years ago

hivemind - 0.9.8: Initial libp2p support, improved DHT protection, better examples

This release contains the following improvements and bugfixes:

• Implement combining validators (#249) (@borzunov)
• Decentralized adaptive optimizers (#243) (@nevec)
• Add nltk to ALBERT example's requirements (#251) (@borzunov)
• Protect training progress and metrics with signatures and DHT schema validation (#250) (@borzunov)
• Add state checkpointing and uploading in coordinator (#241) (@leshanbog @mryab)
• Fix random freezes in averager.step, improve error handling (#254) (@justheuristic @yhn112 @borzunov @mryab)
• Fix device in Switch-MoE, overhaul Server architecture (#256) (@mryab)
• Log more stats for user, move performance stats to examples (#257) (@yhn112)
• Implement authorization for a moderated Hivemind network (#255) (@borzunov)
• Improve error handling, remove deprecated functionality (#261) (@justheuristic @mryab)
• Log correct loss in examples/albert/runfirstpeer.py (#265) (@borzunov)
• Fixed nan when compressing the tensor of zeros (#266) (@Vsevolod-pl)
• Support auxiliary participants in AllReduceProtocol (#260) (@foksly)
• Log collaboration step to Wandb, store metrics only if peer is synchronized (#267) (@borzunov @yhn112 @justheuristic)
• Add initial support for connecting via libp2p (#238) (@MaximKsh @deniskamazur @skobellev @leshanbog @borzunov @mryab @yhn112)

- Python
Published by mryab almost 5 years ago

hivemind - 0.9.7: Improved security, Switch-like MoE, ALBERT example

This release contains the following improvements and bugfixes: * Add RSA signature protection for DHT records (#187) (@borzunov) * Improve Runtime exception handling (#207) (@mryab) * Implement basic decentralized optimizers (#210) (@justheuristic, @mryab) * Add gradient clipping support to ExpertBackend (#214) (@mryab) * Convert SerializerBase to an abstract class (#212) (@mryab) * Replace FeedforwardBlock with a correct implementation (#211) (@mryab) * Disentangle DecentralizedAverager components, add averaging weights (#217) (@justheuristic @mryab) * Add CollaborativeOptimizer, TrainingAverager (#215) (@leshanbog @nevec @mryab) * Move compression-related code to hivemind.utils.compression (#213) (@mryab) * Prevent DecentralizedSGD from accidentally skipping a fraction of training batches (#218) (@ploshkin) * Add uniform compression (#202) (@mponty) * Add gradient buffers to CollaborativeOptimizer (#220) (@justheuristic) * Improve zerograd behavior in CollaborativeOptimizer (#221) (@justheuristic) * Reset gradient buffers when synchronizing with peers (#222) (@justheuristic) * Add tool for custom user experts (#189) (@romakail @justheuristic) * Delta gradients transmission (#225) (@Vsevolod-pl) * Statistics averaging (#229) (@nevec) * Ensure version-consistent result rounding in loadbalance_peers (#230) (@justheuristic @mryab) * Add Switch Transformers-like RemoteMixtureOfExperts (#228) (@mryab) * Add example for collaborative ALBERT training (#226) (@leshanbog @yhn112 @nevec @mryab) * Fix loss metric calculation (#240) (@yhn112) * Add DHT schema validator (#227) (@borzunov) * Fix server hanging in certain cases when connection is lost (#247) (@justheuristic) * Add Dockerfile, refactor tests (#245) (@mryab) * Fix incorrect data types/values in RemoteSwitchMixtureOfExperts (#246) (@mryab)

- Python
Published by mryab about 5 years ago

hivemind - 0.9.6: Patch release

This release adds several new features: * Client-only averaging in AllReduce (#176) * Expert learning rate scheduling (#196) * Quantile compression (#182)

Also, this release contains the following fixes and improvements: * Fix scalar deserialization (#190) * Extract expert-specific methods from DHT (#192)

- Python
Published by mryab about 5 years ago

hivemind - 0.9.5: Patch release

This release fixes several known bugs and security vulnerabilities: * Copytree implementation for py37 compatibility (#162) * Remove pickle.loads in Averager (#160) * Support edge cases for DHT key/subkey/value (#167) * Fix the remaining tests for py37 (#166) * Move Averager metadata serialization out of user scope (#168) * Handle edge cases in DecentralizedAverager (#171) * Fix a typo in quickstart.md (#174) * Serialize DHTID source with msgpack (#172) * Move CLI server launch script to hivemind/hivemind_cli (#173)

- Python
Published by mryab about 5 years ago

hivemind - 0.9.0: Decentralized parameter averaging

• Implement DecentralizedAverager for averaging model parameters & statistics across DHT peers (#119 #123 #134 #140 #141)
• Accelerate RemoteMixtureOfExperts beam search with new key structure (#97 #101 #103 #109)
• Implement lossy compression algorithms for tensors (#102 #106 #112)
• Detect anomalies in RemoteMixtureOfExperts (#132)
• Configure gRPC channels for long-term stability (#129 #131)
• Load expert checkpoints on server startup (#138)
• Support attention mask in example TransformerEncoder layer (#126)
• Add the contribution guide (#156)

Bugfixes: * Fix wrong getattr in hivemind.Server (#122)

Enhancements: * Suport python3.9 and torch1.7 (#142) * Blacklist nonresponsive peers with exponential backoff (#114) * Reuse grpc channels between calls (#120) * Verify DHT peer accessibility and local clock (#137) * Improve logging, remove duplicate log entries (#135) * Improve test coverage (#116)

- Python
Published by mryab about 5 years ago

hivemind - hivemind 0.8.2

• Remove name property from all asyncio tasks (compatibility with python3.7)

- Python
Published by mryab almost 6 years ago

hivemind - hivemind 0.8.1

Minor update: * you can now create minimalistic hivemind server via ./script/runserver.py @Vsevolod-pl * ./script/runserver.py can now sample experts from a pre-defined grid, e.g. expert.[0:256].[0:256] * added quickstart tutorial @justheuristic

- Python
Published by justheuristic almost 6 years ago

hivemind - hivemind 0.8.0

• Speed up tests, shutdown threads in server via threading.Event
• Compile protobuf in setup.py
• Update circleci pipelines
• Update RTD pipeline
• Refactor custom build_ext into install and develop

- Python
Published by mryab almost 6 years ago

hivemind - hivemind 0.7.1

- Python
Published by justheuristic almost 6 years ago