Pembrolizumab in Treating Patients With Advanced Merkel Cell Cancer

Inclusion Criteria: * Patients must have biopsy-proven metastatic MCC or locoregional MCC that has recurred following standard locoregional therapy with surgery and/or radiation therapy * Patients must have measurable disease per Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 assessed by computed tomography (CT) scan, or for skin lesions not measurable by CT scan, measurements may be performed with caliper or flexible ruler * Note: stage IV no evidence of disease (NED) is excluded by this criterion * Have a performance status of =\< 1 on the Eastern Cooperative Oncology Group (ECOG) performance scale * Life expectancy of greater than 6 months * Leukocytes \>= 2,000/mcL (labs should be performed within 14 days of treatment initiation) * Absolute neutrophil count \>= 1,500/mcL (labs should be performed within 14 days of treatment initiation) * Platelets \>= 100,000/mcL (labs should be performed within 14 days of treatment initiation) * Hemoglobin \>= 9 g/dL OR \>= 5.6 mmol/L (labs should be performed within 14 days of treatment initiation) * Serum total bilirubin =\< 1.5 x upper limit of normal (ULN) OR direct bilirubin =\< ULN for patients with total bilirubin levels \> 1.5 x ULN (labs should be performed within 14 days of treatment initiation) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional ULN OR =\< 5 x ULN for patients with liver metastases (labs should be performed within 14 days of treatment initiation) * Serum creatinine =\< 2.5 x ULN OR measured or calculated\* creatinine clearance (CrCl) (glomerular filtration rate \[GFR\] can also be used in place of creatinine or creatinine clearance \[CrCl\]) \>= 30 mL/min for subject with creatinine levels \> 2.5 x institutional ULN * Creatinine clearance should be calculated per institutional standard (labs should be performed within 14 days of treatment initiation) * Thyroid stimulating hormone (TSH) within institutional limits (i.e.: normal); if TSH is greater or less than institutional limits patients may participate if their T4 is within normal limits (WNL); patients may be on a stable dose of replacement thyroid medication; dose adjustments are allowed if needed (labs should be performed within 14 days of treatment initiation) * Patients must provide tissue from an archival tumor sample or newly obtained core, punch or excisional biopsy of a tumor lesion if deemed relatively safe and technically feasible * Note: newly obtained biopsy is preferable * Female patients of childbearing potential must have a negative urine or serum pregnancy within 72 hours before receiving the first dose of study medication; if the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Note: women of child-bearing potential must agree to use 2 methods of birth control, or be surgically sterile, or abstain from heterosexual activity beginning with the screening visit and for the duration of study participation, through 120 days beyond last dose of MK-3475 administration; patients of childbearing potential are those who have not been surgically sterilized or have not been free from menses for \> 1 year; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately * Men treated or enrolled on this protocol must agree to use 2 adequate methods of contraception starting with the screening visit, for the duration of study participation, and through 120 days after the last dose of MK-3475 administration * Ability to understand and willingness to sign a written informed consent document Exclusion Criteria: * Patient has had prior systemic therapy for MCC * Note: prior systemic cytotoxic chemotherapy will be allowed if it was administered in the adjuvant setting (no clinically detectable MCC at the time) and treatment concluded more than 6 months prior to beginning study treatment * Patient is currently participating in or has participated in a study of an investigational systemic agent to treat MCC; or is using an investigational device within 4 weeks of the first dose of treatment * NOTE: if patient received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy * Toxicity from surgery or associated interventions that has not recovered to =\< grade 1 is allowed if it meets the inclusion requirements for laboratory parameters * Patients with locoregional disease that have not received appropriate standard locoregional therapy with surgery and/or radiation therapy * Patient has had radiation therapy within 2 weeks of beginning study treatment * Toxicity from prior radiation therapy has NOT resolved to grade 1 or less * Patient has a diagnosis of immunodeficiency or is receiving systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior the first dose of trial treatment * Patient has had a prior monoclonal antibody for treatment of MCC * Patient has had a prior monoclonal antibody for a non-cancer therapy indication within 4 weeks prior to study day 1 or who has not recovered (i.e., =\< grade 1 or at baseline) from adverse events (AEs) due to agents administered more than 4 weeks earlier * Patient has a known additional malignancy that is progressing or requires active treatment; exceptions include basal cell carcinoma of the skin, squamous cell carcinoma of the skin, or in situ cervical cancer that has undergone potentially curative therapy * Patient has known active central nervous system (CNS) metastases and/or carcinomatous meningitis * Patients with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging for at least 4 weeks prior to the first dose of trial treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids for at least 7 days prior to trial treatment * Patient has a history of allergic reactions attributed to compounds of similar chemical or biologic composition to MK-3475 * Patient has an active autoimmune disease requiring systemic treatment within the past 3 months or a documented history of clinically severe autoimmune disease, or a syndrome that requires systemic steroids or immunosuppressive agents; patients with vitiligo or resolved childhood asthma/atopy would be an exception to this rule; patients that require intermittent use of bronchodilators or local steroid injections would not be excluded from the study; the use of physiologic doses of corticosteroids may be approved after consultation with the protocol principal investigator (PI) and Cancer Immunotherapy Trials Network (CITN); patients with hypothyroidism stable on hormone replacement or Sjogren's syndrome will not be excluded from the study * Patient has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the trial, interfere with the patient's participation for the full duration of the trial, or is not in the best interest of the patient to participate, in the opinion of the treating investigator * Patient has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial * Patient has uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, interstitial lung disease, non-infectious pneumonitis, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Patient is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 120 days after the last dose of trial treatment * NOTE: pregnant women are excluded from this study; breastfeeding should be discontinued if the mother is treated with MK-3475 * Men and nonpregnant, non-breast-feeding women may be enrolled if they are willing to use 2 methods of birth control or are considered highly unlikely to conceive; highly unlikely to conceive is defined as (1) surgically sterilized, or (2) postmenopausal (a woman who is \>= 45 years of age and has not had menses for greater than 1 year will be considered postmenopausal), or (3) not heterosexually active for the duration of the study; the 2 birth control methods can be barrier method or a barrier method plus a hormonal method to prevent pregnancy; subjects should start using birth control from the screening visit throughout the study period up to 120 days after the last dose of study therapy * The following are considered adequate barrier methods of contraception: diaphragm, condom (by the partner), copper intrauterine device, sponge, or spermicide; appropriate hormonal contraceptives will include any registered and marketed contraceptive agent that contains an estrogen and/or a progestational agent (including oral, subcutaneous, intrauterine, or intramuscular agents) * Patients should be informed that taking the study medication may involve unknown risks to the fetus (unborn baby) if pregnancy were to occur during the study; in order to participate in the study, they must adhere to the contraception requirement (described above) for the duration of the study and during the follow-up period described above; if there is any question that a subject will not reliably comply with the requirements for contraception, that subject should not be entered into the study * Pregnancy: if a patient inadvertently becomes pregnant while on treatment with MK-3475, the patient will immediately be removed from the study; the site will contact the patient at least monthly and document the patient's status until the pregnancy has been completed or terminated; the outcome of the pregnancy will be reported without delay and within 24 hours if the outcome is a serious adverse experience (e.g., death, abortion, congenital anomaly, or other disabling or life-threatening complication to the mother or newborn); the study investigator will make every effort to obtain permission to follow the outcome of the pregnancy and report the condition of the fetus or newborn; if a male patient impregnates his female partner the study personnel at the site must be informed immediately and the pregnancy reported and followed * Subjects who are breast-feeding are not eligible for enrollment * Patient is human immunodeficiency virus (HIV) positive Note: patients who are human immunodeficiency virus (HIV) positive may participate IF they meet the following eligibility requirements: * They must be stable on their anti-retroviral regimen, and they must be healthy from an HIV perspective * They must have a cluster of differentiation (CD)4 count of greater than 250 cells/mcL * They must not be receiving prophylactic therapy for an opportunistic infection * Patient has known active hepatitis B (e.g., hepatitis B surface antigen \[HBsAg\] reactive) or hepatitis C (e.g., hepatitis C virus \[HCV\] ribonucleic acid \[RNA\] \[qualitative\] is detected) * Note: a positive hepatitis B serology indicative of previous immunization (i.e., hepatitis B surface antibody \[HBsAb\]-positive and hepatitis B core antibody \[HBcAb\]-negative) or a fully resolved acute hepatitis B infection is not an exclusion criterion * Has active non-infectious pneumonitis \>= grade 2; or history of grade 3 non-infectious pneumonitis within the past 12 months; or any history of grade 4 non-infectious pneumonitis * History of other pulmonary disease such as emphysema or chronic obstructive pulmonary disease (COPD), (forced expiratory volume in 1 second \[FEV1\] \< 60% of predicted for height and age); pulmonary function tests (PFTs) are required in patients with prolonged smoking history or symptoms of respiratory dysfunction * Cardiovascular disease that meets one of the following: congestive heart failure (New York Heart Association class III or IV), active angina pectoris, or recent myocardial infarction (within the last 6 months) * Prior organ allograft or allogeneic transplantation, if the transplanted tissue is still in place * Patient has had live vaccines within 30 days before the first dose of trial treatment and while participating in the trial; examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, chicken pox, yellow fever, seasonal flu, H1N1 flu, rabies, bacille de Calmette et Guérin (BCG), and typhoid vaccine