Vaccine Therapy and Sargramostim in Treating Patients With Pancreas Cancer That Cannot Be Removed By Surgery

Inclusion Criteria: * Patients must have histologically or cytologically confirmed pancreatic adenocarcinoma * Patients may have locally advanced disease, not amenable to curative resection; the site of pancreatic cancer must be amenable to endoscopic ultrasound (EUS) injection; patients with newly diagnosed metastatic disease of small volume may be included in the study at the investigator's discretion; such patients would be limited to those with: * Liver involvement \< 10% of volume and no metastasis \> 2 cm, and/or * Pulmonary involvement with no respiratory compromise and no metastasis \> 2cm and/or * Peritoneal disease and no metastasis \> 2 cm and without ascites (as might be found on exploratory laparoscopy) * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 80%) * Patients (in the opinion of the principal investigator) should be able to complete a full 3-month course of vaccination preferably with an anticipated survival of 6 months or longer * Leukocytes \>= 3,000/mcL * Hemoglobin \>= 8 gms/dL * Absolute neutrophil count \>= 1,500/mcL * Platelets \>= 100,000/mcL * Total bilirubin =\< 1.5 X institutional upper limit of normal * Aspartate aminotransferase (AST)(serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT)(serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 X institutional upper limit of normal * Prothrombin time (PT)/partial thromboplastin time (PTT) within normal institutional limits * Amylase/lipase =\< 1.5 X institutional upper limit of normal * Creatinine =\< 1.5 X institutional upper limit of normal * Urine Protein =\< grade 1 or 24-hour urine protein =\< 1000 mg for patients with proteinuria above 1+ * Urinalysis: No evidence of casts * The effects of PANVAC-V (vaccinia) and/or PANVAC-F (fowlpox) on the developing human fetus are unknown; for this reason, women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for four months following the last vaccine dose; should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately * Patients may not have any other active illness (e.g., uncontrolled infection, uncontrolled cardiac disease) that would preclude safe therapy * Patients must sign a written informed consent document Exclusion Criteria: * Patients may not have had radiotherapy to the pancreas * Patients who have had chemotherapy or radiotherapy within 4 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study or those who have not recovered from adverse events due to agents administered more than 4 weeks earlier * Patients may not be receiving nor have received any other investigational agents within 28 days prior to registration * Patients with known brain metastases should be excluded from this clinical trial * History of allergic reactions or severe adverse reactions attributed to compounds of similar chemical or biologic composition to PANVAC-V (vaccinia) and/or PANVAC-F (fowlpox) which include but are not limited to the viral vectors vaccinia (small pox vaccination) and fowlpox, allergy to GM-CSF or to eggs which are used for the production of the vaccine * Systemic corticosteroid therapy within 28 days of registration; topical steroids, steroid eye drops or inhaled steroids are contraindicated for at least 2 weeks before vaccinia vaccination and at least 4 weeks post vaccinia vaccination * Uncontrolled intercurrent illness including, but not limited to active infection, symptomatic congestive heart failure or documented cardiomyopathy, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements; patients with symptomatic cardiac disease or congestive heart failure who are not stable on current medications or have significant impairment of function such as class III New York Heart Association (NYHA), recent cardiac events including myocardial infarction or cerebrovascular accident within six months of entry, and/or unstable or uncontrolled arrhythmia or angina * Active pancreatitis defined as clinically symptomatic hyperamylasemia and/or hyperlipasemia * Pregnant women are excluded from this study; breast-feeding should not occur for at least 4 months following completion of therapy with the recombinant vaccine * Human immunodeficiency virus (HIV)-positive patients and patients with hepatitis B and C are ineligible because of likely reduced immune competence which could affect the ability to respond to the vaccine * Evidence of immunodeficiency or immune suppression; autoimmune diseases such as the following: autoimmune neutropenia, thrombocytopenia, or hemolytic anemia, systemic lupus erythematosus, Sjogren's syndrome, or scleroderma, myasthenia gravis, Goodpasture's syndrome, Addison's disease, Hashimoto's thyroiditis, or active Graves' disease * Prior or concurrent extensive eczema or acute, chronic, or exfoliative skin disorders (e.g., extensive psoriasis, burns, impetigo, or disseminated zoster, varicella zoster, severe acne, or other open rashes or wounds) * Unable to avoid close contact or household contact with the following high-risk individuals for 3 weeks after the PANVAC-V (vaccinia) vaccination: * Children under the age of 3 year * Pregnant or nursing women * Individuals with active or a history of eczema or atopic dermatitis, or Darier's disease; those with other acute, chronic or exfoliative skin conditions (e.g., burns, impetigo, varicella zoster, severe acne, contact dermatitis, psoriasis, herpes or other open rashes or wounds) until the condition resolves * Immunocompromised individuals (by disease or therapy) such as those with acquired immune deficiency syndrome (AIDS) * Concurrent malignancy (i.e., malignancy other than adenocarcinoma of the pancreas), unless the subject has been curatively treated and disease free for \>= 2 years, except non-melanoma skin cancer or in-situ cervical cancer * Splenectomy