microcitation-parser

Parse microcitations

https://github.com/rdmpage/microcitation-parser

Science Score: 31.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
○
.zenodo.json file
○
DOI references
○
Academic publication links
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (12.6%) to scientific vocabulary

Last synced: 10 months ago · JSON representation ·

Repository

Parse microcitations

Basic Info

Host: GitHub
Owner: rdmpage
Language: PHP
Default Branch: main
Size: 7.38 MB

Statistics

Stars: 0
Watchers: 2
Forks: 0
Open Issues: 0
Releases: 0

Created about 3 years ago · Last pushed about 1 year ago

Metadata Files

Readme Citation

Microcitation parser

Tools to parse "microcitations" to literature in the Biodiversity Heritage Library (BHL) and elsewhere.

SQLite and Heroku

Because of the size of the SQLite database it is managed using LFS. This posses problems for Heroku as it doesn’t recognise LFS by default. To get this to work we do the following:

Add https://github.com/infinitly/heroku-buildpack-git-lfs as a build pack, see infinitly/heroku-buildpack-git-lfs
We create a personal access token for GitHub, see https://github.com/settings/tokens
We add this repository to the project as the value for the environment variable (AKA config key) GIT_LFS_REPOSITORY as https://<token>@github.com/rdmpage/<repository>.git.

Now Heroku will load the database file using LFS and build the app.

BHL database

In the data directory there is a simple JSON file that stores titles and the corresponding BHL TitleID. There his no attempt to clean or normalise the titles, we simply match what we have. In the development version we can add new matches to this file.

There is also a SQLite database of BHL data. The table title is generated by fetching the file title.txt from BHL, extracting just the three columns that we need using php parse-titles.php > titles.tsv then using php import-titles.php to load that into SQLite.

ISSN database

The data/issn directory has several TSV files listing journal names and ISSNs. These can be used to create a local database to look up ISSNs.

Works database

The data/works directory has several TSV files listing basic metadata for “works” (i.e., articles). These can be used to create a local database to look up DOIs based on simple metadata.

SQL to generate TSV file

SELECT issn, series, volume, issue, spage, epage, year, doi FROM publications_doi WHERE issn IN ("0960-4286","1474-0036");

SQL to update records with missing data

If we subsequently update the external source database(s), for example, by adding series numbers or pagination, we can generate SQL update statements to move this to the database used by this service.

Series

SELECT DISTINCT "UPDATE works SET series=""" || series || """ WHERE volume=""" || volume || """ AND issue=""" || issue || """ AND year=""" || year || """ AND issn=""" || issn || """;" FROM publications_doi WHERE issn='0374-5481' AND series IS NOT NULL;

spage

SELECT DISTINCT "UPDATE works SET spage=""" || spage || """ WHERE doi=""" || doi || """;" FROM publications_doi WHERE issn='1175-5334' AND spage IS NOT NULL;

epage

SELECT DISTINCT "UPDATE works SET epage=""" || epage || """ WHERE doi=""" || doi || """;" FROM publications_doi WHERE issn='1175-5334' AND epage IS NOT NULL;

Owner

Name: Roderic Page
Login: rdmpage
Kind: user
Location: Glasgow
Company: University of Glasgow

Website: http://iphylo.blogspot.com
Twitter: rdmpage
Repositories: 399
Profile: https://github.com/rdmpage

Citation (citation.php)

<?php

// Microparser that matches to BHL

require_once (dirname(__FILE__) . '/lib/api_utilities.php');
require_once (dirname(__FILE__) . '/lib/pid_utilities.php');
require_once (dirname(__FILE__) . '/lib/microparser.php');
require_once (dirname(__FILE__) . '/site.php');

// Web site to document API

class MySite extends Site
{
	// Styles specific to this site
	function style_custom()
	{
		echo '.data {
			margin-top:1em;
			padding:1em;
			border:1px solid rgb(222,222,222);
			border-radius: 0.2em;
			font-family:monospace;
			white-space:pre-wrap;			
			height:300px;	
			overflow-y:auto;	
		}

		';	
	}	
	
	// Javascript specific to <HEAD> for this site
	function javascript_head()
	{
	
	}	
	
	// Javascript specific to <BODY> for this site
	function javascript_body_start()
	{
		echo '<script>
		function go() {
			var output = document.getElementById("output");
			//output.style.display = "none";
			output.innerHTML = "<progress></progress>";
			
			var citation = document.getElementById("citation");
			citation.style.display = "none";

			var q = document.getElementById("q").value;
			
			var doc = {};
			doc.q = q;
			
			var url = "' . $_SERVER['PHP_SELF'] . '";
			
			fetch(url, {
				method: "post",
				body: JSON.stringify(doc)
			}).then(
					function(response){
						if (response.status != 200) {
							console.log("Looks like there was a problem. Status Code: " + response.status);
        					return;
						}
				
						response.json().then(function(data) {
						
						output.innerHTML = JSON.stringify(data, null, 2);							
						output.style.display = "block";
						
						
						if (data.data.citation) {
							
							var html = "<h2>Formatted citation</h2>";
							
							
							if (data.data.DOI) {
								html += "<a href=\"https://doi.org/" + data.data.DOI + "\" target=\"_new\">";
							}
							
							html += data.data.citation;
							
							if (data.data.DOI) {
								html += "</a>";
							}
														
							citation.innerHTML = html;
							citation.style.display = "block";
						}
					
					});
				}
			);
		}	
	</script>';
	}	
	
	// Display home page, or error message if badness happened
	function default($error_msg = "")
	{	
		$this->start();
		$this->header();
		$this->main_start();
		
		if ($error_msg != '')
		{
			echo '<div><strong>Error!</strong> ' . $error_msg . '</div>';
		}
		else
		{
			// main content	
			echo '<h1>Parse microcitation and get formatted citation</h1>';
			
			$example = "Aust. J. Zool. 6: 280 [key], 281, figs 2G, 2H, 3D, pl. 1, fig. 2.";
			
			echo '<p>This tool takes a "microcitation" such as "' . $example . '",
			attempts to convert it into structured data, looks for a DOI, and if found, the DOI
			is used to add a formatted citation.
			</p>';

			echo '<p>To use as an API simply GET the site URL with the citation as the parameter "q", 
			e.g.</p>
			<p> <a href="' . $_SERVER['PHP_SELF'] . '?q=' . urlencode($example) . '">?q=' . $example . '</a></p>
			<p>
			or POST a JSON document of the form <code>{ "q": "microcitation" }</code>. The API will return results as JSON.
			</p>';
			
			
		echo '
		<div class="flexbox" >
				<div class="stretch">
					<input type="input" id="q" name="q" placeholder="Citation..." value="' . $example . '" autofocus/>
				</div>
				<div class="normal">
					<button onclick="go()">Go</button>
				</div>
		</div>	
		';	
			
			echo '<div id="output" class="data"></div>';
			
			echo '<div id="citation" style="display:none;"></div>';
		}
		
		$this->main_end();
		$this->footer();
		$this->end();	
	}
}



// Web or API?
if (count($_GET) == 0 && $_SERVER['REQUEST_METHOD'] != 'POST')
{
	// Web
	$site = new MySite("Parse microcitation and get formatted citation");
	$site->default();
}
else
{
	// API
	$doc = null;

	if ($_SERVER['REQUEST_METHOD'] == 'GET')
	{
		$doc = http_get_endpoint(["q"]);
	}
	else
	{
		$doc = http_post_endpoint(["q"]);
	}

	if ($doc->status == 200)
	{
		$parse_result = parse($doc->q, false);

		// we want to add results to existing doc
		foreach ($parse_result as $k => $v)
		{
			$doc->{$k} = $v;
		}
		
		// BHL specific stuff
		if (isset($doc->data))
		{
			if (isset($doc->data->{'container-title'}))
			{
				$doc = find_issn($doc);
				$doc = find_pid($doc);
								
				if (isset($doc->data->DOI))
				{
					$doc->style = 'apa';
					
					$doc = format_citation($doc);				
				}
			}
		}
		
	}
	
	send_doc($doc);	
}

?>

GitHub Events

Total

Issues event: 1
Issue comment event: 4
Push event: 6

Last Year

Issues event: 1
Issue comment event: 4
Push event: 6

Issues and Pull Requests

Last synced: over 1 year ago

All Time

Total issues: 0
Total pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Total issue authors: 0
Total pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 0
Pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Issue authors: 0
Pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Source Science