https://github.com/christian-konrad/mapreduce-invertedindexer-example
Simplified example of an Inverted Indexer for plain text documents built on Hadoop's MapReduce framework.
https://github.com/christian-konrad/mapreduce-invertedindexer-example
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (9.0%) to scientific vocabulary
Keywords
Repository
Simplified example of an Inverted Indexer for plain text documents built on Hadoop's MapReduce framework.
Basic Info
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
Inverted Indexer built with MapReduce on Hadoop
This inverted indexer simply indexes all word terms (terms with only word-characters and containing min. 1 letter) of plain text files. It builds an index of term: {docName: occurenceCount}.
ℹ The generation of docIDs is omitted as we currently haven't implemented a mapping of those, so we simply return the file names. This is discouraged in general as it is too expensive and unreliable. You could implement a mapping yourself using hadoop's
DistributedCache.
Prerequisitions
Edit the username in config.sh to the one of your HDFS path in the form of /user/$username. The documents to be indexed must be put into the documents directory and will be uploaded to your HDFS file system automatically on run.
⚠ You have to put hadoop to your
PATHvariable to run the scripts or replacehadoopwith the path to your hadoop binary installation.
Run the indexer
You can compile the InvertedIndexer class and make a .jar out of it by simply running ./compile.sh in your shell. Afterwards, run it on your hadoop cluster using ./run.sh. Or, for convenience, do it all together calling ./compile_run.sh.
The inverted index output files can be found afterwards at /user/$username/InvertedIndexer/output.
Customizing the code
The project does not come with any build tool as it is compiled in the hadoop environment. To improve your coding using IntelliSense and others, simply add the following .jars to your project references, depending on the IDE you use. On VS Code, you can add them to your .vscode/settings.json or conveniently using the Java Projects view if you have the corresponding plugin installed.
hadoop-3.x.0\share\hadoop\mapreduce\hadoop-mapreduce-client-core-3.3.0.jarhadoop-3.x.0\share\hadoop\hdfs\hadoop-hdfs-3.3.0.jarhadoop-3.x.0\share\hadoop\common\hadoop-common-3.3.0.jar
Owner
- Name: Christian Konrad
- Login: christian-konrad
- Kind: user
- Location: Frankfurt, Germany
- Company: @camunda
- Website: https://christian-konrad.me/
- Repositories: 9
- Profile: https://github.com/christian-konrad
Product manager and software engineer living near Frankfurt am Main, Germany. My specialization is generalization.
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: 11 months ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0