csci4830-datacenter-pig-assignment
A pig script to join work on patent data
https://github.com/krolitzer/csci4830-datacenter-pig-assignment
Science Score: 18.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
○codemeta.json file
-
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (2.7%) to scientific vocabulary
Last synced: 10 months ago
·
JSON representation
·
Repository
A pig script to join work on patent data
Basic Info
- Host: GitHub
- Owner: krolitzer
- Language: PigLatin
- Default Branch: master
- Size: 97.7 KB
Statistics
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
- Releases: 0
Created over 11 years ago
· Last pushed over 11 years ago
Metadata Files
Readme
Citation
README
A pig script that goes through two patent data sets and finds the percentage of patents from each state that reference patents in the same state. The two files are raw patent data, and a file of patents that reference other patents i.e. (citing,cited) Using joins and other techniques, then end result is achieved.
Owner
- Name: Chris C
- Login: krolitzer
- Kind: user
- Repositories: 12
- Profile: https://github.com/krolitzer
Citation (citationPig-Chris-costello.txt)
-- Citation Project In Pig
-- Christopher Costello 10/14/14
-- DataCenter Scale Computing Fall 2014 - Dirk Grunwald
cites = load 'input/cite75_99.txt' using PigStorage(',') as (citing:long, cited:long);
pats = load 'input/apat63_99.txt' using PigStorage(',')
as (ipatent : long, gyear : int, gdate : int, appyear : int,
country : chararray , postate : chararray,
assignee, asscode,
claims : int, nclass, cat, subcat,
cmade : int, creceive : int, ratiocit : float ,
general, original, fwdaplag, bckgtlag,
selfctub, selfctlb, secdupbd, secdlwbd);
a = filter pats by country == '"US"';
b = foreach a generate ipatent, postate;
j = join b by ipatent, cites by cited;
-- Join together citing cited with state info
halfWay = foreach j generate ipatent, postate, citing;
fullJoin = join halfWay by citing, b by ipatent;
gg = group fullJoin by halfWay::b::postate;
--Count all states
allStates = foreach gg generate group, (double)COUNT(fullJoin);
-- Group and count states that reference themselves
ff = foreach gg generate group, flatten($1);
sameStates = filter ff by fullJoin::halfWay::b::postate == fullJoin::b::postate;
groupedSame = group sameStates by fullJoin::halfWay::b::postate;
countSameStates = foreach groupedSame generate group, (double)COUNT($1);
-- Put the counts together and calculate stats
joinAllCounted = join countSameStates by $0, allStates by $0;
finalStats = foreach joinAllCounted generate $0, ($1 / $3);
dump finalStats;
("AK",0.05461121157323689)
("AL",0.09346744234947782)
("AR",0.06666041627601725)
("AZ",0.10832515079253752)
("CA",0.3162336632039463)
("CO",0.1456790619630854)
("CT",0.1829404208696966)
("DC",0.024247130899991372)
("DE",0.1839289568622614)
("FL",0.1278537584906878)
("GA",0.1810521833031681)
("HI",0.04689165186500888)
("IA",0.11614694574967963)
("ID",0.2907066928235777)
("IL",0.19994720192380216)
("IN",0.17529729688812856)
("KS",0.06912320037762568)
("KY",0.10201486968298278)
("LA",0.13004076878276064)
("MA",0.16267367560160198)
("MD",0.10907652132626415)
("ME",0.07610670236160244)
("MI",0.27695337118802)
("MN",0.2261935305610926)
("MO",0.12727075994565806)
("MS",0.06941500635862653)
("MT",0.13486982501067007)
("NC",0.15682601075723696)
("ND",0.07905444681263321)
("NE",0.08150530717272435)
("NH",0.10927442949093037)
("NJ",0.2043746400123625)
("NM",0.08776738686445233)
("NV",0.1266044340723454)
("NY",0.23578563877167946)
("OH",0.2300836578345342)
("OK",0.2288471518323451)
("OR",0.12947039731737706)
("PA",0.20117256944006656)
("PR",0.0283436669619132)
("RI",0.08970297938759583)
("SC",0.18090672510614983)
("SD",0.0512630014858841)
("TN",0.14286262913322323)
("TX",0.2962074243171029)
("UT",0.1376297712469155)
("VA",0.106569375443577)
("VI",0.016556291390728478)
("VT",0.1300383877159309)
("WA",0.15018422374811588)
("WI",0.1826209573272462)
("WV",0.14371844517425617)
("WY",0.04555925262349629)