Nivolumab With or Without Stereotactic Radiosurgery in Treating Patients With Recurrent, Advanced, or Metastatic Chordoma

Inclusion Criteria: * Patients must have histologically confirmed diagnosis of chordoma; the pathologic confirmation may be from another metastatic site * Patients may have metastases, with newly identified peripheral metastases * Cross-sectional imaging evidence of progression of recurrent or metastatic disease * Previous treatment information (name of agent, treatment starting date, and date of progression) must be available for review * Measurable disease in one or more site. (Per RECIST criteria: Measurable lesions are defined as those that can be accurately measured in at least one dimension (longest diameter to be recorded) as \>= 10 mm with CT scan, MRI, or calipers by clinical exam. All tumor measurements must be recorded in millimeters (or decimal fractions of centimeters)). * Karnofsky performance scale \>= 70% * White blood cells (WBC) \>= 3,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Platelets \>= 100,000/mcL * Total bilirubin =\< 2 x institutional upper limit of normal * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase \[SGPT\]) =\< 2.5 x institutional upper limit of normal * Creatinine within normal institutional limits OR according to Johns Hopkins magnetic resonance imaging (MRI) policy * Women of child bearing potential (WOCBP) should use an adequate method to avoid pregnancy for 5 months plus the time required for nivolumab to undergo approximately five half-lives) after the last dose of investigational drug; in order for a woman to be determined not of child-bearing potential, she must have \>= 12 months of non-therapy induced amenorrhea or be surgically sterile * Men receiving nivolumab and who are sexually active with WOCBP will be instructed to adhere to contraception for a period of 7 months after the last dose of investigational product * Ability to understand and the willingness to sign written informed consent document(s) Exclusion Criteria: * Prior chemotherapy within 21 days or 5 half-lives (whichever is shorter) days of starting treatment * Prior therapy with investigational drugs within 21 days or at least 5 half-lives (whichever is shorter) before study administration * Prior therapy with an anti-PD-1, anti-PD-L1, or anti-PDL-2 antibody * Neurologic dysfunction that would confound the evaluation of neurologic and other adverse events * Known allergy to compounds of similar chemical or biologic composition to nivolumab * Pregnant or breastfeeding women * Known history of human immunodeficiency virus * Active infection requiring therapy, including positive tests for hepatitis B surface antigen or hepatitis C ribonucleic acid (RNA) * Active autoimmune disease, history of autoimmune disease or history of syndrome that required systemic steroids or immunosuppressive medications, e.g. organ, tissue, or allogenic hematopoietic stem cell transplant (HSCT) recipients. Exceptions include those with resolved childhood asthma/atopy. Subjects with asthma who require intermittent use of bronchodilators (such as albuterol) will not be excluded from this study. Subjects are also permitted to enroll if they have vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger * Patients should be excluded if they have a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study drug administration; inhaled or topical steroids and adrenal replacement doses \> 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease * Use of any vaccines against infectious diseases (e.g. varicella, influenza, etc.) up to 4 weeks (28 days) before receiving nivolumab * Prisoners or subjects who are compulsorily detained for treatment of either a psychiatric or physical (e.g. infectious disease) illness * Prior radiation doses equivalent to, or greater than, 8000 cGy in 200 cGy fractions * Any radiation to the target lesions within 6 months of enrollment * Unable to meet radiation treatment plan parameters * Unavailable for follow up visits after treatment * Prior malignancy unless disease free for ≥ 2 years. Curatively treated basal or squamous cell carcinoma of the skin, totally excised melanoma of stage IIA or lower, low- or intermediate-grade localized prostate cancer (Gleason sum ≤7), and curatively-treated carcinoma in situ of the cervix, breast, or bladder are allowed regardless.