Localized Radiation Therapy or Recombinant Interferon Beta and Avelumab With or Without Cellular Adoptive Immunotherapy in Treating Patients With Metastatic Merkel Cell Carcinoma

Inclusion Criteria: * Signed written informed consent * Confirmation of MCC by internal pathology review of initial or subsequent biopsy or other pathologic material * If an accessible lesion is present, a biopsy will be performed within 6 weeks of the start of study intervention; the results of the biopsy must be obtained prior to initiation of study intervention * Evidence of MCPyV TAg tumor expression by immunohistochemistry on any prior or current tumor specimen or viral oncoprotein antibody confirmation within 6 weeks of the start of study intervention * Eastern Cooperative Oncology Group (ECOG) performance status (PS) =\< 2 at trial entry * Patients must have at least one bi-dimensionally measurable lesion by palpation, clinical exam, or radiographic imaging within 6 weeks of the start of study intervention (X-ray, computed tomography \[CT\] scan, positron emission tomography \[PET\] scan, magnetic resonance imaging \[MRI\], or ultrasound) * For patients designated to be treated on Group 2: cardiac ejection fraction \>= 35%; for patients with significant risk factors for coronary artery disease (Framingham risk score \> 15%), a cardiac stress test is recommended * At least 3 weeks must have passed since any of the following: systemic corticosteroids, immunotherapy (for example, T-cell infusions, immunomodulatory agents, interleukins, MCC vaccines, intravenous immunoglobulin, expanded polyclonal tumor infiltrating lymphocytes \[TIL\] or lymphokine-activated killer \[LAK\] therapy), pentoxifylline, other small molecule or chemotherapy cancer treatment, other investigational agents or other systemic agents that target Merkel cell carcinoma Exclusion Criteria: * Known active infections or oral temperature \> 38.2 Celsius (C) fewer than 72 hours prior to receiving study treatment or systemic infection requiring chronic maintenance or suppressive therapy * White blood cells (WBC) \< 200/mcl * Hemoglobin (Hb) \< 8 g/dL * Absolute neutrophil count (ANC) \< 1000/mcl * Platelets \< 50,000/mcl * New York Heart Association functional class III-IV heart failure, symptomatic pericardial effusion, stable or unstable angina, symptoms of coronary artery disease, congestive heart failure, clinically significant hypotension, or history of an ejection fraction of =\< 30 % (echocardiogram or multi gated acquisition scan \[MUGA\]) * Clinically significant pulmonary dysfunction, as determined by medical history and physical exam; patients so identified will undergo pulmonary functions testing and those with forced expiratory volume in 1 second (FEV1) \< 2.0 L or diffusion capacity of the lung for carbon monoxide (DLco) (corrected \[corr\] for hemoglobin \[Hgb\]) \< 50% will be excluded * Creatinine clearance \< 30 ml/min which cannot be attributed to MCC metastasis * Total bilirubin \> 1.5 x upper limit of normal (ULN) * Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) \> 2.5 x ULN; for patients with liver metastases: AST/ALT \> 5 x ULN * Active autoimmune disease (e.g. systemic lupus erythematosus, vasculitis, infiltrating lung disease, inflammatory bowel disease) whose possible progression during treatment would be considered unacceptable by the investigators * Symptomatic and untreated central nervous system (CNS) metastasis; however, patients with 1 to 2 asymptomatic, less than 1 cm brain/CNS metastases without significant edema may be considered for treatment; if sub-centimeter CNS lesions are noted at study entry, then repeat imaging will be performed, if more than 4 weeks have elapsed from the last scan * Any condition or organ toxicity that is deemed by the principal investigator (PI) or the attending physician to place the patient at unacceptable risk for treatment on the protocol * Pregnant women, nursing mothers, men or women of reproductive ability who are unwilling to use effective contraception or abstinence; women of childbearing potential must have a negative pregnancy test within 2-6 weeks prior to treatment * Clinically significant and ongoing immune suppression including, but not limited to, systemic immunosuppressive agents such as cyclosporine or corticosteroids, chronic lymphocytic leukemia (CLL), uncontrolled human immunodeficiency virus (HIV) infection, or solid organ transplantation * Patients may not be on any other treatments for their cancer aside from those included in the protocol; patients may not undergo another form of treatment concurrently with this study * Known severe hypersensitivity reactions to monoclonal antibodies (grade \>= 3 National Cancer Institute \[NCI\]-Common Terminology Criteria for Adverse Events \[CTCAE\] version \[v\] 4.0), any history of anaphylaxis, or uncontrolled asthma * Vaccination with live inactivated viral strains for the prevention of infectious diseases within 4 weeks of the start of the study treatment, inactivated influenza vaccines are permitted while on trial * Known alcohol or drug abuse * Legal incapacity or limited legal capacity