Low Dose Ipilimumab With Pembrolizumab in Treating Patients With Melanoma That Has Spread to the Brain

Inclusion Criteria: * Life expectancy \> 12 weeks. * Subjects must have signed and dated an IRB/IEC (Institutional Review Board/Independent Ethics Committee) approved written informed consent form in accordance with regulatory and institutional guidelines. This must be obtained before the performance of any protocol related procedures that are not part of normal subject care. * Subjects must be willing and able to comply with scheduled visits, treatment schedule, laboratory testing, and other requirements of the study. * Histologically confirmed malignant melanoma with measurable metastases in the brain (\>= 0.5 cm). * At least one measurable intracranial target lesion, which previously was not treated with local therapy (no prior stereotactic radiosurgery \[SRS\] to this lesion). Largest diameter of \>= 0.5 cm, but =\< 3 cm as determined by contrast-enhanced MRI. * Representative formalin-fixed paraffin-embedded (FFPE) tumor specimens in paraffin blocks (blocks are preferred) OR at least 4 unstained slides with an associated pathology report for testing of tumor PD-L1 expression: * Tumor tissue should be of good quality based on total and viable tumor content. * Patients who do not have tissue specimens may undergo a biopsy during the screening period. Acceptable samples include core-needle biopsies for deep tumor tissue or excisional, incisional, punch, or forceps biopsies for cutaneous, subcutaneous, or mucosal lesions. * Tumor tissue from bone metastases is not evaluable for PD-L1 expression and is therefore not acceptable. * However, if repeat biopsy is not feasible, and no archival tissue available patient still may be enrolled * Prior stereotactic radiotherapy (SRT) and prior excision of up to 5 MBM is permitted if there has been complete recovery, with no neurologic sequelae, and measurable lesions remain. Growth or change in a lesion previously irradiated will not be considered measurable. Regrowth in cavity of previously excised lesion will not be considered measurable. Note: Any prior SRT to brain lesions or prior excision must have occurred \>= 2 weeks before the start of dosing for this study. * Prior radiation to non-central nervous system (non-CNS) is allowed, and does not require a washout period for treatment initiation. * Subjects must be free of neurologic signs and symptoms related to metastatic brain lesions and must not have required or received systemic corticosteroid therapy in the 10 days prior to beginning protocol therapy. * ECOG (Eastern Cooperative Oncology Group) performance status =\< 1. * Absolute neutrophil count \>= 1500/uL. * Platelets \>= 100,000/uL. * Hemoglobin \>= 9.0 g/dL or \>= 5.6 mmol/L. * Creatinine =\< 1.5 x upper limit of normal (ULN) OR measured or calculated creatinine clearance \>= 30 L/min with creatinine levels \> 1.5 x institutional ULN. GFR (glomerular filtration rate) can also be used in place of creatinine or CrCl (creatinine clearance). * Total bilirubin =\< 1.5 ULN OR direct bilirubin =\< ULN for participants with total bilirubin levels \> 1.5 x ULN. * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) / alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase \[SGPT\]) =\< 2.5 x ULN (=\< 5 x ULN for participants with liver metastases). * International normalized ratio (INR) OR prothrombin time (PT) and activated partial thromboplastin time (aPTT) =\< 1.5 x ULN unless participant is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use of anticoagulants. * Women of child-bearing potential (WOCBP) must not be breastfeeding and must have a negative pregnancy test within 3 days prior to initiation of dosing. She must agree to use an acceptable method of birth control from the time of the negative pregnancy test up to 120 days after the last dose of study drug. WOCBP must agree to adhere to the contraceptive guidance in the protocol. Note: A female participant is eligible to participate if she is not a woman of childbearing potential as defined by the protocol. * Fertile men must agree to use an acceptable method of birth control as described in the protocol while on study drug and up to 120 days after the last dose of study drug and also refrain from donating sperm during this period. * All associated toxicity from previous or concurrent cancer therapy must be resolved (to =\< grade 1 or baseline) prior to study treatment administration. * Steroids for physiological replacement are allowed. Exclusion Criteria: * History of known leptomeningeal involvement (lumbar puncture not required). * Previous stereotactic or highly conformal radiotherapy within 2 weeks before the start of dosing for this study. Note the stereotactic radiotherapy field must not have included the brain index lesion(s). * Subjects previously treated with SRT \> 5 lesions in the brain * Brain lesion size \> 3 cm. * Prior checkpoint inhibitor therapy. Allowable prior therapy: Approved adjuvant therapies, which may include molecularly-targeted agents, IFN-·, and ipilimumab. * Patients who received ipilimumab as adjuvant or neoadjuvant therapy must have a 6 month washout before receiving any dosing on this study. * Cohort A: Prior anti-PD in the adjuvant setting is allowed, but washout period is 6 months. * For Cohort B: Patients with unresectable metastatic melanoma who received either anti-PD-1 or PDL-1 in the past are eligible. Washout period a minimum 3 weeks. * Subjects with an active, known or suspected autoimmune disease. Subjects with type I diabetes mellitus, hypothyroidism only requiring hormone replacement, skin disorders (such as vitiligo, psoriasis, or alopecia) not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll. * Subjects with major medical, neurologic or psychiatric condition who are judged as unable to fully comply with study therapy or assessments should not be enrolled. * Subject has a history of a second malignancy, unless potentially curative treatment has been completed with no evidence of malignancy for 2 years. Note: The time requirement does not apply to participants who underwent successful treatment of superficial bladder cancer, in situ cervical cancer, or other in-situ cancers. Subjects with a completely treated prior malignancy and no evidence of disease for \>= 2 years are eligible. * Skin cancer exclusion: Please note that basal cell carcinoma and squamous cell carcinoma is exempt from needing resection prior to treatment. (Resection can be completed after the start of treatment). * Has a known history of or is positive for hepatitis B (hepatitis B surface antigen \[HBsAg\] reactive) or hepatitis C (hepatitis C virus \[HCV\] ribonucleic acid (RNA) \[qualitative\] is detected). Note: Without known history, testing needs to be performed to determine eligibility. Hepatitis C antibody (Ab) testing is allowed for screening purposes in countries where HCV RNA is not part of standard of care. * Has a known history of human immunodeficiency virus (HIV) infection. No HIV testing is required unless mandated by local health authority. * The use of corticosteroids is not allowed for 10 days prior to initiation of therapy (based upon 5 times the expected half-life of dexamethasone) except patients who are taking steroids for physiological replacement. If alternative corticosteroid therapy has been used, consultation with the sponsor medical monitor is required to determine the washout period prior to initiating study treatment. * Subjects with a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalent) or other immunosuppressive medications within 14 days of study initiation. Inhaled or topical steroids, and adrenal replacement steroid doses \> 10 mg daily prednisone equivalent, are permitted in the absence of active autoimmune disease. * Subjects with history of life-threatening toxicity related to prior ipilimumab adjuvant therapy except those that are unlikely to re-occur with standard countermeasures (e.g. hormone replacement after adrenal crisis). * Major surgical procedure, open biopsy (excluding skin cancer resection), or significant traumatic injury within 14 days of initiating study drug (unless the wound has healed) or anticipation of the need for major surgery during the study. * Non-healing wound, ulcer, or bone fracture. * Women who are breast-feeding or pregnant. * Uncontrolled intercurrent illness (i.e., active infection \>= grade 2) or concurrent condition that, in the opinion of the Investigator, would interfere with the study endpoints or the subject's ability to participate. * History of clinically significant cardiac disease or congestive heart failure \> New York Heart Association (NYHA) class 2. Subjects must not have unstable angina (anginal symptoms at rest) or new-onset angina within the last 3 months or myocardial infarction within the past 6 months. * Investigational drug use within 14 days (or 5 half-lives, whichever is longer) of the first dose of ipilimumab and pembrolizumab. * Has a history of non-infectious pneumonitis that required steroids or current pneumonitis.