MRI and PET to Assess Pembrolizumab Response

Inclusion Criteria: * Histologically confirmed diagnosis of melanoma * Stage III or stage IV metastatic melanoma * Disease that is measurable. This is defined as lesions measuring at least 10mm on radiologic imaging. For lymph node disease, the lesion must measure at least 15 mm or have been biopsied and shown to contain melanoma. Skin or mucosal lesions that are not measurable on radiologic imaging but measure at least 10mm on clinical exam are also acceptable. (See Section 13.2 for detailed definition of measurable disease) * Disease is termed unresectable or the patient refuses resection * The Eastern Cooperative Oncology Group (ECOG) performance status of 1 or better * Age ≥18 years. This is due to the limited data with pembrolizumab in children younger than 18 years of age. * Normal organ and marrow function as defined below * Aspartate aminotransferase (AST) (SGOT) or alanine aminotransferase (ALT) (SGPT) ≤2.5 × institutional upper limit of normal (≤5 x upper limit of normal for patient with liver metastasis) * Total bilirubin within 1.5 x institutional level of normal or direct bilirubin ≤ upper limit of normal (ULN) for patient with total bilirubin levels \> 1.5 ULN) * Hemoglobin ≥ 9.0g/dL or ≥5.6mmol/L * Absolute neutrophil count ≥1,000/microliters (mcL) * Platelets ≥ 75,000/mcL * The effects of pembrolizumab on the developing human fetus are unknown. For this reason, women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 4 months after completion of pembrolizumab administration. * Women of child-bearing potential must have a negative urinary or serum pregnancy test. Exclusion Criteria: * Prior therapy with anti-PD-1 antibody * The patient has symptomatic brain metastases. Asymptomatic brain metastases are permitted provided that there is no steroid requirement, no more than 4 metastases detected on standard MRI imaging, no metastatic brain lesion that is \> 3 cm in size, and no lepto-meningeal disease. * Patient has not recovered to Grade 0-1 from adverse events due to prior chemotherapy, radiation, or biological cancer therapy (including monoclonal antibody (mAb)). * The patient is not recovered from minor or major surgery and is less than 4 weeks from major surgery prior to starting treatment with pembrolizumab * Significant auto-immune disease requiring hospitalization within the past two years or any history of life-threatening auto-immune disease * Immunosuppressive therapy including systemic corticosteroids except for maintenance dosing for adrenal insufficiency * Concurrent use of any other investigational agents * Active central nervous system metastasis and/or carcinomatous meningitis causing symptoms. * Known additional malignancy that is progressing or requires active treatment with the exceptions of basal cell carcinoma of the skin, squamous cell carcinoma of the skin, or in situ cervical cancer that has undergone potentially curative therapy. * Active autoimmune disease or a syndrome that requires systemic steroids or immunosuppressive agents with the exceptions of replacement dose steroids for adrenal insufficiency, vitiligo, resolved childhood asthma/atopy, intermittent use of inhaled steroids, local steroid injections, hypothyroidism stable on hormone replacement, and Sjogren's syndrome * Active tuberculosis * Hypersensitivity to pembrolizumab or any of its excipients. * Expected to require any other form of systemic antineoplastic therapy while receiving pembrolizumab * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, autoimmune diseases, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Patients with a pacemaker, stainless steel aneurysm clip or any other magnetic resonance (MR) contraindicated implant or foreign body would warrant exclusion from this study. Pacemakers may be reprogrammed or turned off by the strong MRI magnetic field. Radio-frequency (RF) fields in MR can also cause severe heating of pacemaker lead tips. Steel aneurysm clips are prone to torque in the strong MR field which can displace the clips and may damage the vessel, resulting in hemorrhage, and/or death. * All patients and volunteers are screened before MR examination using a MRI safety screening Questionnaire as part of the New York Presbyterian/Columbia University Medical Center (NYP/CUMC) MRI safety policy. Any patient who would normally be excluded by this screening process would also be excluded from this study. * Pregnant or breastfeeding women are excluded from this study because of the unknown effects of the study agent on the unborn child and the possible adverse impact of immune activation on pregnancy. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with pembrolizumab breastfeeding should be discontinued if the mother is treated with pembrolizumab. * Male who is expecting to father a child during the treatment period. * History of pneumonitis requiring hospitalization or systemic immune suppressive therapy. * Significant immunodeficiency making the patient unlikely to benefit from pembrolizumab therapy including a diagnosis of acquired immune deficiency syndrome (AIDS), active hepatitis B or hepatitis C, or organ transplant requiring immunosuppressive therapy. * Received a live virus vaccine within 30 days of planned start of therapy