Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of https://github.com/root-11/tablite

https://github.com/root-11/tablite - 2023.11.6

What's Changed

  • disable nimporter by default by @realratchet in https://github.com/root-11/tablite/pull/166

Full Changelog: https://github.com/root-11/tablite/compare/2023.11.5...2023.11.6

- Python
Published by realratchet about 2 years ago

https://github.com/root-11/tablite - 2023.11.5

What's Changed

  • Respect passed pbars by @realratchet in https://github.com/root-11/tablite/pull/165

Full Changelog: https://github.com/root-11/tablite/compare/2023.11.4...2023.11.5

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.11.4

What's Changed

  • fix text exporter by @realratchet in https://github.com/root-11/tablite/pull/164

Full Changelog: https://github.com/root-11/tablite/compare/2023.11.3...2023.11.4

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.11.3

What's Changed

  • export some apis by @realratchet in https://github.com/root-11/tablite/pull/163

Full Changelog: https://github.com/root-11/tablite/compare/2023.11.2...2023.11.3

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.11.2

What's Changed

  • add support for opening for any number of columns by @realratchet in https://github.com/root-11/tablite/pull/162

Full Changelog: https://github.com/root-11/tablite/compare/2023.11.1...2023.11.2

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.11.1

What's Changed

  • fix timedeltas and non-primitive filter by @realratchet in https://github.com/root-11/tablite/pull/161

Full Changelog: https://github.com/root-11/tablite/compare/2023.11.0...2023.11.1

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.11.0

What's Changed

  • Groupby ported to nim by @omenSi in https://github.com/root-11/tablite/pull/158
  • Forgot to add TQDM to the groupby :) by @omenSi in https://github.com/root-11/tablite/pull/159
  • Fixed some edge cases in filter pagination by @realratchet in https://github.com/root-11/tablite/pull/157
  • Groupby api update by @realratchet in https://github.com/root-11/tablite/pull/160

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.15...2023.11.0

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.15

What's Changed

  • Ported filter to nim by @realratchet in https://github.com/root-11/tablite/pull/156

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.14...2023.10.15

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.14

What's Changed

  • update mplite by @realratchet in https://github.com/root-11/tablite/pull/155

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.13...2023.10.14

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.13

Slowest imputation method (Nearest Neighbor) ported to Nim for speedup by @omenSi and @realratchet in https://github.com/root-11/tablite/pull/154

- Python
Published by omenSi over 2 years ago

https://github.com/root-11/tablite - 2023.10.12

What's Changed

  • Fix potential layout mismatch by @realratchet in https://github.com/root-11/tablite/pull/150

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.11...2023.10.12

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.11

What's Changed

  • fix an issue where imputation does not update page dtypes by @realratchet in https://github.com/root-11/tablite/pull/149

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.10...2023.10.11

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.10

What's Changed

  • fix imputation inconsistency by @realratchet in https://github.com/root-11/tablite/pull/148

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.9...2023.10.10

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.9

What's Changed

  • Column selector API update by @realratchet in https://github.com/root-11/tablite/pull/147

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.8...2023.10.9

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.8

What's Changed

  • Fix imputations by @realratchet in https://github.com/root-11/tablite/pull/146

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.7...2023.10.8

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.7

What's Changed

  • Update some issues with escaped columns by @realratchet in https://github.com/root-11/tablite/pull/145

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.6...2023.10.7

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.6

What's Changed

  • When sheet is not specified, assume first by @realratchet in https://github.com/root-11/tablite/pull/144

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.5...2023.10.6

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.5

What's Changed

  • do not skip headers by @realratchet in https://github.com/root-11/tablite/pull/143

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.4...2023.10.5

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.4

What's Changed

  • Skip empty rows and fixes by @realratchet in https://github.com/root-11/tablite/pull/141

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.3...2023.10.4

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.3

What's Changed

  • get_header fixes by @realratchet in https://github.com/root-11/tablite/pull/140

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.2...2023.10.3

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.2

What's Changed

  • Join changes by @realratchet in https://github.com/root-11/tablite/pull/138

Full Changelog: https://github.com/root-11/tablite/commits/2023.10.2

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.1

What's Changed

  • Fix an issue with pivot_transpose + utils by @realratchet in https://github.com/root-11/tablite/pull/137

Full Changelog: https://github.com/root-11/tablite/compare/2023.10.0...2023.10.1

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.10.0

What's Changed

  • Added nimble support by @realratchet in https://github.com/root-11/tablite/pull/136

Full Changelog: https://github.com/root-11/tablite/compare/2023.9.8...2023.10.0

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Join bug fix

In join we previously had a check that would raise if the input table had no rows. This lead to some data pipelines being disrupted when filtering as an error would be raised. In this patch we have removed this validation, so that an empty table (generating an empty set) can pass.

The logic for this change is that, if this is valid: AA = Table({'A':[1,2,3], 'B':[10,20,30]}) BB = Table({'A1': [1,2,3], 'B1': [1,2,3]}) CC = AA.join(BB, ['B'], ['B1']) CC.show() +===+=====+=====+=====+=====+ | # | A | B | A1 | B1 | |row|empty|empty|empty|empty| +---+-----+-----+-----+-----+ +===+=====+=====+=====+=====+ then so is this:

A = Table({'A':[1,2,3], 'B':[]}) B = Table({'A1': [1]}) C = B.join(A, left_keys=['A1'], right_keys=['B']) C.show() +=+==+=+=+ |#|A1|A|B| +-+--+-+-+ +=+==+=+=+

- Python
Published by root-11 over 2 years ago

https://github.com/root-11/tablite - Fix join issue

Fix join issue when using merge keys

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Bugfixes

Fix an issue where join with merge keys produces an error Fix an issue with nim unpickler and 16bit ints, pickle format always declares them as uint16 Added missing progressbar for filter Reimplemented repagination in nim

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Improved join implementation

Reduced join footprint Added multiprocessing support for join

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Fix some issues

Fixes an issue where we couldn't load numpy scalar pages (produced by tablite.filter), although filter shouldn't produce them in the first place because it's extremely dirty Fix an issue with unicode page slicer Improve exception texts Added missing loader tests

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - minor patch

What's Changed

  • fix minor issue by @realratchet in https://github.com/root-11/tablite/pull/130

Full Changelog: https://github.com/root-11/tablite/compare/2023.9.2...2023.9.3

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - 2023.9.2

What's Changed

  • Added column selector docstrings by @realratchet in https://github.com/root-11/tablite/pull/128
  • Fix an issue with page that is already cast pages by @realratchet in https://github.com/root-11/tablite/pull/129

Full Changelog: https://github.com/root-11/tablite/compare/2023.9.1...2023.9.2

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Minor changes

Fix issue related to rejected column order, tqdm more granular.

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Added column selector

Added column to perform type casting columns to supported types of bool, int, float, string, date, time and datetime.

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Bug fix: Adding tqdm and pbar for sort

Minor bug fix.

Change: - Adding tqdm and pbar for sort

- Python
Published by root-11 over 2 years ago

https://github.com/root-11/tablite - Bugfix for join if pages are fragmented.

This patch fixes #110. No other API changes.

- Python
Published by root-11 over 2 years ago

https://github.com/root-11/tablite - Fix misaligned pages

  • Fix an issue where csvs with mixed layouts (no trailing commas) would get corrupted, they will be treated as empty strings instead.
  • Remove python backend for csv parser as it has been deprecated and superseded by nim

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Improved locale number support

Improved locale number support including mixed locale strings in same column.

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Fix windows PYD file

Fixes windows pyd's that caused them not to load when downloaded as a wheel.

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Test windows pyd

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Nim support for Windows

Windows platform can now use nim backend for text importer to leverage faster import speeds.

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Fix nimporter to support newlines in qualified strings

Fixes the issue where some csvs had newline characters in between qualified (quoted) strings causing segfault when trying to load in numpy because the pages got misaligned.

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Resolve an issue with np.take and <U0 dtype

Fixes an issue caused by a bug in numpy internals on empty strings.

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Fix multi-column import

Fix an issue with corrupt layout when the same column is being imported multiple times.

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Weaker boolean infer in nim

Replaced tight boolean infer to tight boolean infer by ignoring capitalization.

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Patch for join keyword "merge_keys"

Minor patch for keyword "merge_keys" in join.

See https://github.com/root-11/tablite/commit/f1764741b5180a8ab4f9505b5d0552bcec5c5a7e for details

- Python
Published by root-11 over 2 years ago

https://github.com/root-11/tablite - Nim backend + other changes

Nim backend for csv importer. Improve excel importer. Improve slicing consistency. Logical cores re-enabled on *nix based systems. Filter is now type safe. Added merge utility. Various bugfixes.

- Python
Published by realratchet over 2 years ago

https://github.com/root-11/tablite - Adding merge utility to Table and joins

This update features:

New: - join with keyword merge_keys. - Table.merge as a numpy accelerated helper for merging columns.

Fix: - bugfix on excel readers deprecation warning.

- Python
Published by root-11 almost 3 years ago

https://github.com/root-11/tablite - Minor bug fix on pivot transpose

Pivot transpose now raises ValueError when duplicate column names are given as input.

- Python
Published by root-11 almost 3 years ago

https://github.com/root-11/tablite - Fix xlsx headers

Fix issue with xlsx headers

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Environment file & Excel changes

Hidden excel rows/columns will be importer. .env file can be created for development

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Packaging fix

Fix wrong .so being published

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Packaging fix

Packaging fix for nim backend

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Remove nim runtime requirement

Remove nim runtime requirement

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Nim backend for *nix based systems

Text reader will now use nim based backend on *nix based systems.

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - bugfix for statistics for date and time

A bug was detected by dovydasrudys in which statistics for a column wasn't evaluated correctly.

- Python
Published by root-11 almost 3 years ago

https://github.com/root-11/tablite - Typesafe filter

Filter is now type safe, mixed types will use slow per element type comparison but types that are not comparable.

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - String inference fix

Empty strings will no longer become None when using DataType.infer('', str).

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Improved importing via text reader performance

Greatly improved importing text files e.g., csvs, performance. Extremely wide tables will now have linear instead of exponential processing time dependency. Also greatly improved performance per-row performance.

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Enable logical cores again

Enables utilization of logical cores on non-windows based systems.

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Improved slicing consistency

Changes to slicing behavavior to improve consistency when a table contains only a single column.

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Feature: offset header row

+added ability to offset header row when importing headerrowindex *fixed an issue with get_headers for text formats when only one column was available

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Fixes for xls get_headers inconsistencies

Fix issues with get_headers falling back to text reading when reading 0 lines of excel, fix issue where reading excel file would ignore file count, excel file reader now has parity for linecount selection.

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Fixed get_headers linecount issue

Fixed a logic bug in get_headers that caused one extra line to be returned than requested.

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Fix refcount for deep table copies

Updated the way reference counting works. Tablite now tracks references to used pages and cleans them up based on number of references to those pages in the current process. This change allows to handle deep table clones when sending tables via processes (pickling/unpickling), whereas previous implementation would corrupt all tables using same pages due to reference counting asserting that all tables are shallow copies to the same object.

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - Update mplite dependency

Updated mplite dependency, changed to soft version requirement to prevent pipeline freezes due to small bugfixes in mplite.

- Python
Published by realratchet almost 3 years ago

https://github.com/root-11/tablite - 2023.6.1 - Breaking release with ~6x speed up

This release 2023.6.1 addresses #61 with a significant number of changes and features ~6x speedup. This release note summarizes the changes from 2023.6.dev1 through 2023.6.dev14

For a quick summary view the tutorial

Changes

A new Config-class available in config.py, which sets: - Config.pagesize = 1000000 - Config.workdir = tempfile.gettempdir() - Config.DISABLETQDM: as a boolean switch to silence tqdm.

The workdir uses ///pages for managing pages, so that different python processes don't interact with the same storage location. Previously two python processes would enter conflict on the same tablite.h5-file. If the python process is sigkill'ed, the temp folder will loiter. If the python process exits (sigint) python will clean up the temp folders.

Table now accepts the keyword columns as a dict in init: t = Table(columns={'b':[4,5,6], 'c':[7,8,9]})` Table now accepts header/data combinations in init: t = Table(header=['b','c'], data=[[4,5,6],[7,8,9]])`

With these features it is no longer necessary to write: python t = Table t['b'] = [4,5,6] t['c'] = [7,8,9]

Preferred approach for subclassing tables is: ```python class MyTable(tablite.Table): def init(self, args, *kwargs): super().init(args, *kwargs) self.x = kwargs.get("x", 42) # <== special variable required on MyTable.

def copy(self):
    # tablite.Table implements
    # new = cls();   # self.x is now 42 !!!
    # for name,column in self.items():
    #     new[name] = column

    # MyTable therefore implements:
    cp = super(MyTable, self).copy()
    cp.x = self.x  # updating to real x.
    return cp

```

Replace is refactored from using a single value to a mapping, 

    example:
    >>> t = Table(columns={'A': [1,2,3,4]})
    >>> t['A'].replace({2:20,4:40})
    >>> t[:]
    np.ndarray([1,20,3,40])
  • performance benchmarks have been added.
  • Table.head has been moved to tablite.tools where it belongs.

  • histogram now returns two lists instead of a dict as python treats True and 1 as identical.

  • Table.save now defaults to zipfile.ZIPDEFLATED, compressionlevel=1 as this is only 10% slower, yet saves 80% disk space.

  • Column.iter_by_page : An iterator for traversing data by page.

  • module reindex implements a constant memory method for re-indexing tasks which now is used by Lookup, Join, Sort and Filter.

  • New function unique_index which allows drop_duplicates and sortation to run with constant memory footprint.

  • get_headers now has text_qualifier to match importing

  • Table.types() is practically instantaneous as type information is kept on Pages. dev9 feature Table.dtypes() is deprecated as it almost duplicates the functionality of .types

  • multiprocessing has been disabled for algorithms that guarantee a constant memory footprint.

Deprecated

The following assignment method is DEPRECATED: python t = Table() t[('b','c')] = [ [4,5,6], [7,8,9] ] Which then produced the table with two columns: t['b'] == [4,5,6] t['c'] == [7,8,9]

  • copy_to_clipboard and copy_from_clipboard as pyperclip doesn't seem to be maintained.
  • class method from_dict as Table(columns=dict) now is supported.
  • Insert and append have been removed to discourage operations that are IO-intensive.
  • Column.insert is being removed as it encourages the user to use slow operations. It is better to perform the data manipulation in memory and drop the result into the column using col.extend(....) or col[a:b] = [result].
  • reload_saved_tables. Use t.save(path) and Table.load(path) instead.
  • reset_storage is deprecated as all tables not explicitly saved are considered temporary/volatile.
  • from_dict is deprecated as Table(columns={dict}, ...) is accepted.
  • to_numpy. Default table['name'] returns a numpy array. User should call table['name'].tolist() to get python lists (up to 6x slower) than just retrieving the numpy array.

Everything else remains.

- Python
Published by root-11 almost 3 years ago

https://github.com/root-11/tablite - Bugfix for join & filter

The is a bugfix release. No api changes.

- Python
Published by root-11 about 3 years ago

https://github.com/root-11/tablite - Bugfix for sort

No api changes. This release solves bug #71

- Python
Published by root-11 about 3 years ago

https://github.com/root-11/tablite - Table.dtypes removed.

omission in previous RC.

- Python
Published by root-11 about 3 years ago

https://github.com/root-11/tablite - instant Table.types()

  • Table.types() is practically instantaneous as type information is kept on Pages
  • Table.dtypes() is deprecated as it almost duplicates the functionality of .types

- Python
Published by root-11 about 3 years ago

https://github.com/root-11/tablite - get_headers improvements

get_headers now has text_qualifier to match importing

- Python
Published by realratchet about 3 years ago

https://github.com/root-11/tablite - Maintenance release

No API changes. New function unique_index which allows drop_duplicates and sortation to run with constant memory footprint.

Benchmarks for filter, unique and drop_duplicates confirm this.

- Python
Published by root-11 about 3 years ago

https://github.com/root-11/tablite - Bugfix for redux.filter_all

Minor bugfix release.

- Python
Published by root-11 about 3 years ago

https://github.com/root-11/tablite - 2023.6.dev7 minor bugfixes

  • Fixed an issue with repagination.
  • Fixed couple issues with empty table datatype resolution

- Python
Published by realratchet about 3 years ago

https://github.com/root-11/tablite - 2023.6.dev6 - constant memory footprint

New:

  • Config.DISABLE_TQDM: A boolean switch to silence tqdm.
  • Table.save now defaults to zipfile.ZIPDEFLATED, compressionlevel=1 as this is only 10% slower, yet saves 80% disk space.
  • Column.iter_by_page : An iterator for traversing data by page.
  • module reindex implements a constant memory method for re-indexing tasks which now is used by Lookup, Join, Sort and Filter.
  • Benchmarks have been updated

- Python
Published by root-11 about 3 years ago

https://github.com/root-11/tablite - 2023.6.dev5 - adding Table.dtypes and benchmarks

This release solves https://github.com/root-11/tablite/issues/61#issuecomment-1589058788 and adds performance benchmarks

- Python
Published by root-11 about 3 years ago

https://github.com/root-11/tablite - 2023.6.dev4 release

Sortation passes through tqdm as per previous tablite version

- Python
Published by realratchet about 3 years ago

https://github.com/root-11/tablite - 2023.6.dev3 release

contains bug-fixes for sort (mixed types with text was sorted incorrectly). histogram now returns two lists instead of a dict as python treats true and 1 identically.

- Python
Published by root-11 about 3 years ago

https://github.com/root-11/tablite - 2023.6.dev2 development release

Development release

Contains:

  • Various bugfixes.
  • Multiprocessing is disabled until we can calculate memory footprint to prevent multi process memory crashes.

- Python
Published by root-11 about 3 years ago

https://github.com/root-11/tablite - 2023.6.dev0 - Breaking release with ~6x speedup.

This release 2023.6.dev0 addresses #61 with a significant number of changes and features ~6x speedup.

For a quick summary view the tutorial

Changes

A new Config-class available in config.py, which sets: - config.pagesize = 1000_000 - config.workdir = tempfile.gettempdir()

The workdir uses ///pages for managing pages, so that different python processes don't interact with the same storage location. Previously two python processes would enter conflict on the same tablite.h5-file.

Table now accepts the keyword columns as a dict in init: t = Table(columns={'b':[4,5,6], 'c':[7,8,9]})` Table now accepts header/data combinations in init: t = Table(header=['b','c'], data=[[4,5,6],[7,8,9]])`

With these features it is no longer necessary to write: python t = Table t['b'] = [4,5,6] t['c'] = [7,8,9]

Preferred approach for subclassing tables is: ```python class MyTable(tablite.Table): def init(self, args, *kwargs): super().init(args, *kwargs) self.x = kwargs.get("x", 42) # <== special variable required on MyTable.

def copy(self):
    # tablite.Table implements
    # new = cls();   # self.x is now 42 !!!
    # for name,column in self.items():
    #     new[name] = column

    # MyTable therefore implements:
    cp = super(MyTable, self).copy()
    cp.x = self.x  # updating to real x.
    return cp

```

Replace is refactored from using a single value to a mapping, 

    example:
    >>> t = Table(columns={'A': [1,2,3,4]})
    >>> t['A'].replace({2:20,4:40})
    >>> t[:]
    np.ndarray([1,20,3,40])

Column.histogram() will now return a dict with {key1: count1, key2:count2,...} instead of two lists people then have to zip into a dict anyway.

Table.head has been moved to tablite.tools where it belongs.

Deprecated

The following assignment method is DEPRECATED: python t = Table() t[('b','c')] = [ [4,5,6], [7,8,9] ] Which then produced the table with two columns: t['b'] == [4,5,6] t['c'] == [7,8,9]

  • copy_to_clipboard and copy_from_clipboard as pyperclip doesn't seem to be maintained.
  • class method from_dict as Table(columns=dict) now is supported.
  • Insert and append have been removed to discourage operations that are IO-intensive.
  • Column.insert is being removed as it encourages the user to use slow operations. It is better to perform the data manipulation in memory and drop the result into the column using col.extend(....) or col[a:b] = [result].
  • reload_saved_tables. Use t.save(path) and Table.load(path) instead.
  • reset_storage is deprecated as all tables not explicitly saved are considered temporary/volatile.
  • from_dict is deprecated as Table(columns={dict}, ...) is accepted.
  • to_numpy. Default table['name'] returns a numpy array. User should call table['name'].tolist() to get python lists (up to 6x slower) than just retrieving the numpy array.

Everything else remains.

- Python
Published by root-11 about 3 years ago

https://github.com/root-11/tablite - Fixed a memory leak in filter

Fixed a SharedMemory leak in filter function.

- Python
Published by realratchet about 3 years ago

https://github.com/root-11/tablite - Multiprocessing join fix, other minor changes

copy, filter, sort, any, all methods now properly respects the table subclass.
Filter for tables with under SINGLE_PROCESSING_LIMIT rows will run on same process to reduce overhead.
Errors within child processes now properly propagate to parent.
Table.reset_storage(include_imports=True) now allows the user to reset the storage but exclude any imported files by setting include_imports=False during Table.reset(...).
Bug: A column with 1,None,2 would be written to csv & tsv as "1,None,2". Now it is written "1,,2" where None means absent.
Fix mp join producing mismatched columns lengths when different table lengths are used as an input or when join product is longer than the input table.

- Python
Published by realratchet about 3 years ago

https://github.com/root-11/tablite - Lookup fixes + minor changes

  1. Table.load now properly subclassess the table instead of always resulting in tablite.Table.
  2. Table.from_* methods now respect subclassess, fixed some from_* methods which were instance methods and not class methods.
  3. Fixed Table.from_dict only accepting list and tuple but not tablite.Column which is an equally valid type.
  4. Fix lookup parity in single process and multiple process outputs.
  5. Fix an issue with multiprocess lookup where no matches would throw instead of producing None.
  6. Fix an issue with filtering an empty table.

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Join changes

Changed join to process 1M rows per task to avoid potential OOM on lower memory systems.
Added mp_merge_columns to MemoryManager that merges column pages into a single column.
Fix join parity in single process and multiple process outputs.
Fix an issue with multiprocess join where no matches would throw instead of producing None.

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Upgrade mplite

What's Changed

  • Bumps mplite with deadlock fix by @realratchet in https://github.com/root-11/tablite/pull/55

Full Changelog: https://github.com/root-11/tablite/compare/2022.11.14...2022.11.15

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Improve locking mechanism

Improve locking mechanism to allow retries when opening file as the previous solution could cause deadlocks when running multiple threads.

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Empty page fix

fix empty page value error when creating an empty table with memory manager

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Auto-create workdir

Tablite now creates it's work directory if path is missing.

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - text_reader improvements

text_reader tqdm tracks the entire process now. text_reader properly respects free memory in *nix based systems. text_reader no longer discriminates against hyperthreaded cores.

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Improve get_headers performance on excel files

get_headers now uses plain openpyxl instead of pyexcel wrapper to speed up fetch times ~10x on certain files.

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Safe failure for unknown characters

get_headers and get_delimiter will ignore invalid encoded characters instead of throwing an exception

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Fix a bug with task size calculation on single core systems.

Fix a bug with task size calculation on single core systems.

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Added tablite temporary directory routing

  • Added TABLITE_TMPDIR environment variable for setting tablite work directory.
  • Characters that fail to be read text reader due to improper encoding will be skipped.
  • Fixed an issue where single column text files with no column delimiters would be imported as empty tables.

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Date inference fix

Fixes some issues with datetime and date inference

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Fix for negative slicing

Fixes column[-1] slicing issue producing incorrect results

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Transpose changes

API changes: * table.transpose(...) was renamed to table.pivot_transpose(...) + new table.transpose() and table.T were added, it's functionality acts similarly to numpy.T, the column headers are used the first row in the table when transposing.

- Python
Published by realratchet over 3 years ago

https://github.com/root-11/tablite - Bugfix for non-ascii characters during `.add_rows`

Minor bugfix release. See release notes for details. Credits for the discovery go to cerv15 !

- Python
Published by root-11 over 3 years ago

https://github.com/root-11/tablite - bugfix for CSV reading: default encoding now utf-8 as this includes ascii

Filereader utils would try to determine the encoding using the first 10,000 characters when importing a file. Should a CSV contain a ascii up till the 10k threshold but still contain utf-8 characters, the file reader would raise encoding error.

However as utf-8 is backward compatible with ascii, it is more robust to default to utf-8 instead of ascii.

- Python
Published by root-11 over 3 years ago

https://github.com/root-11/tablite - Bugfix for datatypes.infer

Previously datatypes.infer(1, float) would raise value error instead of 1.0

Credits: Audrius Kulikajevas

- Python
Published by root-11 over 3 years ago

https://github.com/root-11/tablite - 2022.11.0 release

New table features:

  • Table.diff(other, columns=...),
  • table.removeduplicatesrows(),
  • table.drop_na(*arg),
  • table.replace(target,replacement),
  • table.imputation(sources, targets, methods=...),
  • table.topandas() and Table.frompandas(pd.DataFrame),
  • table.to_dict(columns, slice),
  • Table.from_dict(),
  • table.transpose(columns, keep, ...),

New column features:

  • Column.count(item),
  • Column[:] is guaranteed to return a python list.
  • Column.to_numpy(slice) returns np.ndarray.

new tools library: from tablite import tools includes:

  • date_range(start,end),
  • xround(value, multiple, up=None), and,
  • guess as short-cut for Datatypes.guess(...).

bugfixes:

  • __eq__ was updated but missed __ne__.
  • in operator in the filter would crash if both datatypes were not strings.

deferred

  • The memory manager is being profiled for further speedups and was dropped from 2022.11.0. This feature will be included soon.

- Python
Published by root-11 over 3 years ago