Pembrolizumab and Capecitabine in Treating Patients With Locally Advanced or Metastatic Triple Negative or Hormone-Refractory Breast Cancer That Cannot Be Removed by Surgery

Inclusion Criteria: * Patients must have histologically-confirmed unresectable, locally advanced or metastatic breast cancer that meets one of the following: * Triple negative, defined as estrogen receptor (ER) negative, progesterone receptor (PR) negative, human epidermal growth factor receptor 2 (HER2) negative; HER2 negative defined as immunohistochemistry (IHC) 0 or 1+ or fluorescence in situ hybridization (FISH) negative * Hormone-refractory breast cancer which denotes progression to endocrine therapy (e.g., tamoxifen, aromatase inhibitors, fulvestrant) unless contraindicated * Patients must have an Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 * Patients must have a life expectancy of \>= 90 days * Patients must have baseline laboratory tests within the following parameters at least 4 weeks (28 days) prior to registration: * Absolute neutrophil count (ANC) \>= 1,500/mcL * Platelets \>= 100,000/mcL * Hemoglobin \>= 9 g/dL or \>= 5.6 mmol/L without transfusion or erythropoietin (EPO) dependency * Serum creatinine =\< 1.5 x upper limit of normal (ULN) OR measured or calculated creatinine clearance (GFR can also be used in place of creatinine or CrCl) \>= 60 mL/min for subject with creatinine levels \> 1.5 x institutional ULN * Serum total bilirubin =\< 1.5 x ULN OR direct bilirubin =\< ULN for subjects with total bilirubin levels \> 1.5 ULN * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x ULN OR =\< 5 x ULN for subjects with liver metastases * Albumin \>= 2.5 mg/dL * International normalized ratio (INR) or prothrombin time (PT) =\< 1.5 x ULN unless subject is receiving anticoagulant therapy as long as PT or partial thromboplastin time (PTT) is within therapeutic range of intended use of anticoagulants * Activated partial thromboplastin time (aPTT) =\< 1.5 X ULN unless subject is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use of anticoagulants * Females of child-bearing potential (FOCBP) must have a negative serum or urine pregnancy test within 7 days prior to registration and must be at least within 3 days prior to first dose of study drug. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * (Note: a FOCBP is any woman \[regardless of sexual orientation, having undergone a tubal ligation, or remaining celibate by choice\] who meets the following criteria: * Has not undergone a hysterectomy or bilateral oophorectomy * Has had menses at any time in the preceding 12 consecutive months (and therefore has not been naturally postmenopausal for \> 12 months) * Female subjects of childbearing potential (FOCBP) must be willing to use an adequate method of contraception; contraception must be used for the course of the study through 120 days after the last dose of study medication * Note: abstinence is acceptable if this is the usual lifestyle and preferred contraception for the subject * Male subjects of childbearing potential must agree to use an adequate method of contraception; contraception must be used starting with the first dose of study therapy through 120 days after the last dose of study therapy * Note: abstinence is acceptable if this is the usual lifestyle and preferred contraception for the subject * Patients must have the ability to understand and the willingness to sign a written informed consent prior to registration on study * Patients must be willing and able to comply with scheduled visits, treatment plan and laboratory tests * Patient must be able to swallow and retain oral medication Exclusion Criteria: * Patients with documented HER2-positive metastatic disease are not eligible, even if their primary breast cancer was HER2-negative * Patients who have received prior anti-cancer therapy (e.g., biologic or other targeted therapy, chemotherapy) within 2 weeks prior to registration; hormone therapy is permitted until registration * Note: patients who received prior anti-PD-1, PD-L1 or PD-L2 agents are still eligible * Patients who have not recovered from adverse events to grade 1 severity or lower due to agents administered more than 2 weeks earlier than registration, are not eligible, except for stable sensory neuropathy (=\< grade 2) and alopecia * Patients who have received radiotherapy =\< 4 weeks prior to registration, with the exception of palliative radiotherapy, who have not recovered from side effects of such therapy to baseline or grade =\< 1 are not eligible for participation * Patients with central nervous system (CNS) involvement may participate if they meet all the following criteria: * At least 4 weeks from prior therapy completion (including radiation and/or surgery) to starting the study treatment, * Clinically stable with respect to the CNS tumor at the time of screening * Patients who have undergone major surgery =\< 4 weeks prior to registration or have not recovered from side effects of such procedure are not eligible for participation * Patients may not be receiving any other investigational agents * Patients who have a history of allergic reactions or hypersensitivity reactions attributed to compounds of similar chemical or biologic composition to pembrolizumab and/or humanized antibodies are not eligible * Known hypersensitivity to capecitabine, fluorouracil, or any component of the formulation * Note: prior capecitabine is permitted * History of autoimmune disease, including but not limited to myasthenia gravis, myositis, autoimmune hepatitis, systemic lupus erythematosus, rheumatoid arthritis, inflammatory bowel disease, vascular thrombosis associated with antiphospholipid syndrome, Wegener's granulomatosis, Sjogren's syndrome, Guillain-Barre syndrome, multiple sclerosis, vasculitis or glomerulonephritis * Patients with history of autoimmune-related hypothyroidism on a stable dose of thyroid replacement hormone may be eligible for this study * Patients with controlled type I diabetes mellitus on a stable dose of insulin regimen for more than a month may be eligible for this study * Patients who have evidence of active, noninfectious pneumonitis or have a history of severe pneumonitis that required treatment with steroids are not eligible for this study; (Note: replacement physiologic dose of steroids \[prednisone 10 mg daily or equivalent\] are allowed) * Patients who have an uncontrolled intercurrent illness including, but not limited to any of the following, are not eligible: * Hypertension that is not controlled on medication (defined as \>= 140/100 at rest, average of 3 consecutive readings) * Ongoing or active infection requiring systemic treatment * Symptomatic congestive heart failure * Unstable angina pectoris * Cardiac arrhythmia * Clinically significant electrocardiogram (ECG) abnormality, e.g., a repeated demonstration of a QTc interval \> 500 ms * Psychiatric illness/social situations that would limit compliance with study requirements * Known positive test for human immunodeficiency virus (HIV) * Has known active hepatitis B (e.g., hepatitis B surface antigen \[HBsAg\] reactive) or hepatitis C (e.g., hepatitis C virus \[HCV\] ribonucleic acid \[RNA\] \[qualitative\] is detected) * Active tuberculosis * Prior allogeneic bone marrow transplantation or solid organ transplant * Administration of a live, attenuated vaccine within 4 weeks before starting the study treatment or anticipation that a live attenuated vaccine will be required during the study * Any other illness or condition that the treating investigator feels would interfere with study compliance or would compromise the patient's safety or study endpoints * Female patients who are pregnant or nursing (lactating) are not eligible * Patients exhibiting any other condition that would, in the Investigator's judgment, preclude patient's participation in the clinical study due to safety concerns or compliance with clinical study procedures are not eligible for participation; this might include, but is not limited to, infection/inflammation, intestinal obstruction, and/or social/psychological complications * Patients with impaired gastrointestinal (GI) function or GI disease that may significantly alter absorption of oral capecitabine (e.g. ulcerative disease, uncontrolled nausea, vomiting, diarrhea, malabsorption syndrome or small bowel resection) are not eligible for participation * Patients with a history of another malignancy that progressed or required treatment within 5 years prior to registration are not eligible for participation; Note: the exceptions to this include non-melanoma skin cancer or excised carcinoma in situ of the cervix