Stereotactic Body Radiation Therapy With or Without Nivolumab in Treating Patients With Stage I-IIA or Recurrent Non-small Cell Lung Cancer

Inclusion Criteria: * Histological confirmation of non-small cell lung cancer (NSCLC) by either biopsy or cytology is required for the primary diagnosis and is recommended for recurrent disease. The following primary cancer types are eligible: squamous cell carcinoma, adenocarcinoma (with or without bronchioloalveolar carcinoma features), large cell carcinoma (with or without neuroendocrine features), neuroendocrine carcinoma (either NSCLC with neuroendocrine features or atypical carcinoids, but not small cell lung carcinoma), bronchioloalveolar cell carcinoma, or non-small cell carcinoma not otherwise specified * Stage I or selected stage IIa according to the 7th version of the International Association for the Study of Lung Cancer (IASLC) system: stage I (T1 or T2a \[tumor size =\< 5 cm\] N0M0) stage IIa (T2 \[tumor size \> 5 cm but =\< 7 cm\] N0M0) * Patients with multiple primary lung tumors (defined below) are eligible: * Synchronous tumors (diagnosed within 6 months \[mo\]), * Different histology, * Same histology, * Second tumor in different lobed or lung; * Metachronous tumors (diagnosed \> 6 mo apart), * Different histology, * Same histology, * Second tumor in different lobe or lung, * Tumor-free interval of at least 4 years (y) * Patient with multiple primary lung tumors who will receive multiple courses of stereotactic body radiation therapy (SBRT) (50Gy in 4Fx or 70Gy in 10Fx) are eligible * Patients with isolated lung parenchymal recurrent/persistent NSCLC (histology as defined in eligibility criterion 1) after prior definitive surgery or radiotherapy/chemotherapy, when the lesion or lesions are suitable for SABR, are also eligible. * Patients with a single metastatic focus in the lung parenchyma with no other lesions are also eligible, because this presentation is challenging to distinguish from recurrent disease. * Recurrent disease can be in the same lobe or a different lobe but should not invade critical structures (esophagus, brachial plexus, major vessels, heart, spinal cord); should not involve any lymph node; and should not include any other suspicious lesions in the lung or any other locations. * Any prior therapy (surgery, radiotherapy, or systemic) must have been completed at least 12 weeks before administration of the study drug. * Tumors should be =\< 7 cm (measured by CT imaging in the lung window setting) with N0M0; PET) imaging is required for restaging (per eligibility criterion) and any lymph node suspected of harboring tumor should be confirmed by biopsy (per eligibility criterion) * Both Chest CT and PET/CT are required within 16 weeks and at least one of them needs to be done within 12 weeks (plus/minus 1 week). Any lymph node suspected of harboring disease based on its shape, size, or PET standardized uptake value (SUV) should be discussed by treating physician and diagnostic radiologist * Patients with medically inoperable stage I disease (T1 or T2a \[tumor size =\< 5 cm\] N0M0) or selected stage IIa disease (T2 \[tumor size \> 5 cm but =\< 7 cm\] N0M0) who have poor lung function or other significant cardiovascular or other comorbidity such as diabetes are eligible. Patients with operable disease who choose to have SABR are also eligible. * The standard justification for medical inoperability is based on pulmonary function and can include any of the following: baseline forced expiratory volume in 1 second (FEV1) \< 50% of predicted value; diffusion capacity \< 50% of predicted value; baseline hypoxemia or hypercapnia; exercise oxygen consumption \< 50% of predicted value; severe pulmonary hypertension; severe cerebral, cardiac, or peripheral vascular disease; and severe chronic heart disease * Patients must have a Zubrod performance status score of 0-2 (2 is included here because such patients are typically \> 70 years old and cannot tolerate surgery) * The following mandatory staging studies must be done within 12 weeks before study registration: * Chest CT or PET/CT scans of both lungs, the mediastinum, adrenal glands and rest of the body; primary tumor dimension will be measured on diagnostic CT and again on simulation CT using the lung window setting. * Mediastinoscopy or endobronchial ultrasound-guided biopsy of the mediastinal lymph nodes is recommended and required for any patients with PET/CT or CT findings suggesting lymph node involvement. * Brain magnetic resonance imaging (MRI) or CT scan if symptoms or signs suggest brain metastases. * Invasive mediastinal staging: For all patients with CT or PET evidence of hilar involvement (level 10) or with mediastinal lymph nodes \> 1.0 cm in the shortest diameter or clinically suspicious by treating physician and/or radiologist, disease must be staged by cervical mediastinoscopy, esophageal endoscopic ultrasound-guided biopsy, or endobronchial ultrasound-guided biopsy * For patients with left-sided tumors and enlarged nodes (\> 1.0 cm in the shortest diameter) on the aortopulmonary window setting, the aortopulmonary nodes must be biopsied by extended mediastinoscopy, Chamberlain procedure, video-assisted thoracoscopic surgery (VATS) approach, or ultrasound-guided biopsy to ensure that the patient does not have N2 disease. At the time of cervical mediastinoscopy, esophageal endoscopic ultrasound-guided biopsy, or endobronchial ultrasound-guided biopsy, the following nodal stations must be examined and biopsied, if present: * Ipsilateral nodal station 4 * Contralateral nodal station level 4, and * Subcarinal nodes (level 7) For left-sided tumors, any lymph node in the superior or anterior mediastinum \> 1.0 cm in the shortest axis on CT or positive on PET must be identified and biopsied. Eligibility requires that any PET-positive mediastinal or distant sites must be biopsy-negative. However, if the treating physician and diagnostic radiologist strongly suspect PET positivity, the patient should not be enrolled even if the biopsy is negative (to exclude patients with false-negative biopsy) * All patients must sign a study-specific consent form * All patients (men \& women) of childbearing potential should use a method of birth control that is effective for them throughout their participation in this study. * Women of childbearing potential should use an adequate contraceptive method to avoid pregnancy for 5 months (30 days plus the time required for nivolumab to undergo five half-lives) after the last dose of investigational drug; must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) within 24 hours before the start of nivolumab; and must not be breastfeeding. * Men who are sexually active with women of childbearing potential must use a contraceptive method with a failure rate of less than 1% per year; men receiving nivolumab will be instructed to use contraception for 7 months after the last dose of nivolumab. * Women who are not of childbearing potential (i.e. are postmenopausal or surgically sterile) \& azoospermic men do not require contraception * White blood cell (WBC) \>= 2000/uL (within 30 days before study registration) * Neutrophils (Neuts) \>= 1500/uL (within 30 days before study registration) * Platelets (PLT) \>= 100 x 10\^3/uL (within 30 days before study registration) * Hemoglobin (HGB) \> 9.0 g/dL (within 30 days before study registration) * Serum creatinine =\< 2 x upper limit of normal (ULN) or creatinine clearance (calculated with the Cockcroft-Gault formula) \>= 30 mL/min (within 30 days before study registration) * Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 3 x ULN (within 30 days before study registration) * Total bilirubin (Bili T) =\< 1.5 x ULN (although patients with Gilbert syndrome can have total bilirubin \< 3.0 mg/dL (within 30 days before study registration) Exclusion Criteria: * Patients with tumors \> 7 cm or tumors involving the main bronchus or associated vessels or tumors that invade any critical structures (such as esophagus, brachial plexus, heart, mediastinal major vessels) are not suitable for SABR * Patients with direct evidence of regional or distant metastases after appropriate staging studies, or with synchronous non-lung primary or prior non-lung malignancy (other than nonmelanomatous skin cancer or in situ cancer) diagnosed within the past 3 years are not eligible. Patients with a history of curable non-lung cancer up to 3 years before registration and have been cancer-free for 2 years are eligible * Patients who have received previous immunotherapy with PD1 or CTLA4 antibodies are not eligible * Patients with plans to receive other concomitant local therapy (including standard fractionated radiotherapy and surgery) or other systemic therapy (including chemotherapy, target therapy and other type of immunotherapy or investigative agents) while on this protocol, except at disease progression, are not eligible * Female patients who are pregnant or lactating are not eligible (because treatment involves unforeseeable risks to the embryo or fetus) * Patients for whom SABR plans cannot meet the minimum requirement of target coverage and dose-volume constraints of critical structures (see SABR treatment planning section) are not eligible * Patients who have active, known, or suspected autoimmune disease are not eligible. However, patients with vitiligo, type I diabetes mellitus, residual hypothyroidism due to an autoimmune condition that requires only hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll * Patients with a known history of antibodies to human immunodeficiency virus (HIV) -1 or -2 are not eligible. Patients with live vaccines * Patients with a known positive test for hepatitis B virus surface antigen (HBV sAg) or hepatitis C virus ribonucleic acid (HCV antibody) indicating acute or chronic infection are not eligible * Patients who have a condition requiring systemic treatment with corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study drug administration are not eligible. However, inhaled or topical steroids, adrenal replacement doses, and \> 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease * Patients with allergies or adverse drug reactions to the following are not eligible: * History of allergy to study drug components; * History of severe hypersensitivity reaction to any monoclonal antibody * Patients who have had prior treatment with an anti-PD1, anti-PDL1, anti-PDL2, anti-CTLA4 antibody, or any other antibody or drug specifically targeting T-cell costimulation or immune checkpoint pathways are not eligible * Patients are known to have contraindications to radiotherapy